Trying to design something that doesn't yet exist can be daunting and a set methodology can really help make such a task seem more manageable. Let me start by saying that there is no right or wrong way to go about it. If you have found a process that works for you, I encourage you to keep using it. For those of us who are just starting into designing a game and are already feeling overwhelmed by the thought of creating something new out of thin air, I wrote this in a step-by-step style that I hope will help.
Step 1: Theme
As a personal preference, I like to start with theme. What is you game going to be about? How do you want your players to feel? What is the scenario you are trying to portray? To me, this all-important aspect of a game will dictate everything from game mechanics to style of art. Arguably, theme isn't required in all games. Abstract games, such as the traditional game of "Go", have no theme and still plays well. The lack of a theme can make a game more competitive and perhaps appeal to a wider audience, and it is up to you whether you want to make an abstract game or not. Personally, I always like to have a theme.
Step 2: Mechanics
Mechanics are perhaps the most important part of a game. This is how the players will play the game, how they win, how they lose, how they form strategy. This is what makes a game fun! There are a lot of game mechanics you can utilize in existing games and you can even create your own to fit with your theme! You can start thinking about mechanics and then choose a theme, however, mashing your theme with mechanics that don't fit can be frustrating to players and cause confusion so it's always good to keep in mind the experience you are trying to create for the player.
Step 3: Rules
How does one play your game? Rules are important so that everyone knows how to play. Rules prevent unfair game play that can quickly turn a fun game into a frustrating mess. You have a theme and mechanics that you think work well together; now it's time to draft the rules of your game! Here are some sections to include in your first draft to ensure a comprehensive guide for your players:
Introduction: You can choose to include this or not. Personally, this helps to keep my theme in mind. This might be a short narrative about the characters in your game or a little tidbit of the era...anything that helps to keep the mood and experience you're trying to convey in mind! It doesn't have to make it to your final draft.
Objective: What are your players trying to accomplish? How do they win? Explain in one sentence.
Components: You might not know what pieces you need yet, and that's okay! If you have some idea of what components your game will require though, it's good to include them.
Component Types: Maybe you have different kinds of cards, maybe every pawn does something different, or maybe you have special dice; this is a good place to explain the differences between components
Setup: Try to explain to a first-time player how to set up your game so that they can play it. Explain in words as if the person you're addressing has never set up a tabletop game in their life. I recommend also using examples and pictures to illustrate how the play surface should look like after setup.
Game Play: How do you play the game? Explain how to play your game in words as if the person you're addressing has never played a game before. What does the first player do on their first turn? Do all other players do this too? Use steps and bullets and examples.
How to Win: Elaborate more on how you win the game. Your objective gave a general idea of what your goal is as a player; now it's time to explain how you win, what to do in event of a tie, scoring, and any special circumstances that can affect a player's victory.
Credits: You don't need to include this but I recommend it. As you go about designing your game, others tend to lend a helping hand and it's good to keep track of these people. Maybe someone offered to edit your rule book, illustrate your cards, or demo your game at a convention. That's not to say you have to include every single person who played your game in the credits section of your rule book, but it's nice to give acknowledgement where it's due.
Step 4: Components List
More often than not, I already have components in mind while I'm writing the rules of my game but if you haven't thought of what pieces you need to make your game playable, this would be a good time (and don't forget to go back and update the components section of your rules!). This is when I open up a new Excel document and start listing every single component needed for this game to work. I list the type of component, what effects it may have, and the quantity needed as well. Components may be added or removed as you playtest. You'll find that your game might play better without certain components, or you'll want to try adding something to your game to see how it plays, either way, this document will be changing often.
Once you have your components list where you think it will work, it's time to prototype and test it! I wrote last week about making your first prototype. You can read that article here: Tools and Methods to Creating Your First Tabletop Prototype