Phew!
Thoughts
Finally I managed to install PyTorch from source following the docs and lots of articles from medium that I sadly didn't save. Thankfully I had perplexity pro which I used a lot and helped me tremendously. Still, for that niggling issue - I had to get my hand dirty and scour debug log and internet for the fixes.
Journal
- Ansible playbook that I wrote helped me a lot to breeze past the initial required stuff.
- Followed installation document from pytorch repo in github.
- While building pytorch, I crashed my PC cause apparently 64GB is not enough for this hungry monster.
- Found a way that MAX_JOBS=5 is the parameter I can set to have this issue resolved.
- Monitored my memory and CPU usage. CPU usage was 100% intermittenly but memory stuck around 20GB, sometimes 30GB.
- I tried setting up to 10 but it instantly shot up my memory usage, so stuck with 5 and went with the build.
- It ran for an hour maybe, I just left it and slept like a log. When I came back - I found finally that it installed and it worked!
- It worked! Wow!
Learnings
- General - Humbled by Linux. Just installed bunch of stuff. Not really.
- Related to AI - Nah.