Use markdown to write more readable documentation

.gitignore

@@ -1 +1,2 @@
-*__pycache__*
+*__pycache__*
+.vscode

README.md

@@ -2,62 +2,53 @@
 
 ## What is rocket-learn?
 
-rocket-learn is a machine learning framework specifically designed for Rocket League Reinforcement Learning. 
-It works in conjunction with Rocket League, RLGym, and Bakkesmod.
+rocket-learn is a machine learning framework specifically designed for Rocket League Reinforcement Learning. It works in conjunction with Rocket League, [RLGym](https://rlgym.org/), and [Bakkesmod](https://bakkesmod.com/).
 
 ## What features does rocket-learn have?
 
-<ul>
-<li>Reinforcement learning algorithm available out of the box</li>
-  <ul>
-    <li>Proximal Policy Optimization (PPO)</li>
-    <li>extensible format allows new algorithms to be added</li>
-  </ul>
-<li>Distributed compute from multiple computers</li>
-<li>Automatic saving of and training against previous agent versions</li>
-<li>Trueskill progress tracking</li>
-<li>Training against Hardcoded/Pretrained Agents</li>
-<li>Training against Humans</li>
-<li>Saving and loading models</li>
-<li>wandb logging</li>
-</ul>
-
+- Reinforcement learning algorithm available out of the box
+  - Proximal Policy Optimization (PPO)
+  - Extensible format allows new algorithms to be added
+- Distributed compute from multiple computers
+- Automatic saving of and training against previous agent versions
+- Trueskill progress tracking
+- Training against Hardcoded/Pretrained Agents
+- Training against Humans
+- Saving and loading models
+- wandb logging
 
 ## Should I use rocket-learn?
 
-You should use Stable Baselines3 (SB3) to make your bot at first. The hardest parts of building a 
-machine learning bot are 
+You should use [Stable Baselines3 (SB3)](https://stable-baselines3.readthedocs.io/en/master/) to make your bot at first. The hardest parts of building a machine learning bot are:
 
-- understanding how to program
-- understanding how machine learning works
-- choosing good hyperparameters
-- choosing good reward functions
-- choosing an action parser
-- making a statesetter that puts the bot in the best situations
+- Understanding how to program
+- Understanding how machine learning works
+- Choosing good hyperparameters
+- Choosing good reward functions
+- Choosing an action parser
+- Making a statesetter that puts the bot in the best situations
 
-SB3 is a great way to figure out those essential parts. Once you have all of those aspects down, rocket-learn
-may be a good next step to a better machine learning bot. 
+SB3 is a great way to figure out those essential parts. Once you have all of those aspects down, rocket-learn may be a good next step to a better machine learning bot.
 
-If you *don't* yet have these, rocket-learn will add a large amount of complexity for no added benefit. It's 
-important to remember that high compute and a tough opponent are less important than good fundamentals of ML.
+If you *don't* yet have these, rocket-learn will add a large amount of complexity for no added benefit. It's important to remember that high compute and a tough opponent are less important than good fundamentals of ML.
 
 ## How do I setup rocket-learn?
 
-1) Get [Redis](https://docs.servicestack.net/install-redis-windows) running 
+1) Get [Redis](https://docs.servicestack.net/install-redis-windows) running
 
-*__Improper Redis setup can leave your computer extremely vulnerable to Bad Guys. 
-We are not responsible for your computer's safety. We assume you know what you are doing.__*
+> [!WARNING]
+> Improper Redis setup can leave your computer extremely vulnerable to Bad Guys. We are not responsible for your computer's safety. We assume you know what you are doing.
 
-2) Clone the repo
+1) Clone the repo
 
-```
+```shell
 git clone https://github.com/Rolv-Arild/rocket-learn.git
 ```
 
 3) Start up, in order:
 
-- the Redis server
-- the Learner
-- the Workers
+- The Redis server
+- The Learner
+- The Workers
 
-Look at the examples to get up and running
+Look at the examples to get up and running.

docs/README.md

@@ -0,0 +1,5 @@
+# rocket-learn documentation
+
+- [Network Setup](network_setup_readme.md)
+- [SB3 to rocket-learn Transition](sb3_to_rocketlearn_transition.md)
+- [Troubleshooting](troubleshooting.md)

docs/network_setup_readme.md

@@ -0,0 +1,61 @@
+# NETWORK INPUT
+
+rocket-learn expects both actor and critic networks to have an input dimension equal to observation length.
+If the observation outputs an array of size (1, 150) (note the batch dimension of the observation output), then the network input should be 150. As an example:
+
+```python
+actor = DiscretePolicy(Sequential(
+  Linear(150, 256),
+  ReLU(),
+  Linear(256, 256),
+  ReLU(),
+  Linear(256, total_output),
+  SplitLayer(splits=split)
+), split)
+```
+
+# NETWORK OUTPUT
+
+rocket-learn expects actor networks to output a set of probabilities for each possible action. For example, the default Discrete Action allows `8` actions, `5` of which are discrete control choices and `3` of which are boolean choices. Because the Discrete control choices can each be `-1`, `0`, or `1` and each boolean can be `True` or `False`, the network must output `((5 * 3) + (3 * 2))` aka `21` total actions. The actions must then be split into properly sized groups for each actions.
+
+```python
+split = (3, 3, 3, 3, 3, 2, 2, 2)
+total_output = sum(split)
+
+class SplitLayer(nn.Module):
+    def __init__(self, splits=(3, 3, 3, 3, 3, 2, 2, 2)):
+        super().__init__()
+        self.splits = splits
+
+    def forward(self, x):
+        return torch.split(x, self.splits, dim=-1)
+
+actor = DiscretePolicy(nn.Sequential(
+    nn.Linear(INPUT_SIZE, 256),
+    nn.ReLU(),
+    nn.Linear(256, total_output),
+    SplitLayer(split)
+), split)
+```
+
+As another example, KBM actions allow `2` Discrete controls and `3` boolean controls so the network must output `((2 * 3) + (3 * 2))` aka `12` total actions
+
+```python
+split = (3, 3, 2, 2, 2)
+total_output = sum(split)
+
+class SplitLayer(nn.Module):
+    def __init__(self, splits=(3, 3, 3, 3, 3, 2, 2, 2)):
+        super().__init__()
+        self.splits = splits
+
+    def forward(self, x):
+        return torch.split(x, self.splits, dim=-1)
+
+actor = DiscretePolicy(nn.Sequential(
+    nn.Linear(INPUT_SIZE, 256),
+    nn.ReLU(),
+    nn.Linear(256, total_output),
+    SplitLayer(split)
+), split)
+```

docs/sb3_to_rocketlearn_transition.md

@@ -0,0 +1,13 @@
+# SWITCHING FROM [STABLE-BASELINES3](https://stable-baselines3.readthedocs.io/en/master/) TO ROCKET-LEARN
+
+When you install rocket-learn, verify that the version of [rlgym](https://rlgym.org/) is compatible with rocket-learn. Use only the release version of rlgym. Do not use beta of rlgym unless you are attempting to beta test. When setting up your environment, make sure you do not install rlgym from a cached version on your machine, and verify that the dll in the [bakkesmod](https://bakkesmod.com/) plugin folder is accurate.
+
+SB3 abstracts away several important parts of ML training that rocket-learn does not.
+
+- Your rewards will not be normalized
+- Your networks will not have orthogonal initialization by default (assuming you use PPO)
+
+This can drastically affect the results you get and it is not uncommon to not see the same results
+in rocket-learn as you did in SB3, at least until you make tweaks. In addition to the major
+differences listed above, differences in implementation in learning algorithms can cause large
+changes in results. Be prepared to do some extra tweaking as a part of the switch.

docs/troubleshooting.md

@@ -0,0 +1,32 @@
+# TROUBLESHOOTING COMMON PROBLEMS
+
+### RuntimeError: mat1 and mat2 shapes cannot be multiplied (AxB and CxD)
+
+Compare the actor and critic input size to the observation size. They need to be identical. Remember that changing team size and self play can change the observation size.
+
+### Blue Screen of Death error (related to wandb)
+
+1) Rollback your Nvidia driver to WHQL certified September 2021
+
+OR 2) Comment out the following lines of code in the wandb repo:
+
+```python
+try:
+    pynvml.nvmlInit()
+    self.gpu_count = pynvml.nvmlDeviceGetCount()
+except pynvml.NVMLError:
+```
+
+### Can't get Redis to work
+
+- WSL2 is probably the easiest way to get things working.
+- Double check that you can ping the redis server locally.
+
+### There are no errors but changes I'm making don't seem to be affecting anything
+
+- Double check that observations, rewards, and action parsers are the same on both the learner and workers.
+
+### wandb is not working properly or giving you hint errors in your IDE
+
+- Check that there isn’t a folder created called wandb in your project.
+- This is created automatically by wandb, you can change the name in the init call so it doesn’t interfere.