Its a good question but it has a complex (IMHO) answer.
First - the forensic challenges on HTB aren’t always aligned to real-world forensic practices. Its more a case that they use forensic techniques to a varying degree. I will use forget-me-not as an example (as it has retired). Yes the extraction of the string uses what could be described as memory forensics but thats about it.
So on to malware analysis. The vast majority of this uses some reverse engineering techniques but the goals are different which means the workflows can be different.
I would say that Malware Analysis is a different field to Reverse Engineering but there is overlap (for example, SANS has a course called Reverse Engineering Malware: FOR610: Reverse Engineering Malware Training | Malware Tools & Techniques | SANS Institute just to muddy the waters).
It is worth noting that malware analysis ranges from trying to get some simple IOCs to use for the incident response process through to understanding the malware enough to describe what it does.
Reverse engineering is normally focused on finding out what the application does (and in the context here, working out how to exploit it). As you can see this is part of what a malware analyst needs to do and probably aligns with the needs of more advanced practitioners.
Also, a vast majority of the time, malware analysis is looking at malicious documents and scripts rather than pulling apart the internals of a PE/ELF/Mach-O file. A malware analyst is likely to be using oletools as much as they use gdb. At a basic level, malware analysis can be as simple as dropping a file into PEStudio - that gets a massive amount of the information you need for DFIR.
As a reverse engineer, you need a deeper understanding of the file, the ability to walk through what it is doing etc.
So, tl;dr - they are very similar. A good reverse engineer will be good at malware analysis. A good malware analyst will probably have learned reverse engineering.
@tXxc said:
Should I first start with some basics of RE and the dive deeper into MA?
I dont think it matters. If you start with RE of executables, the main MA you’d need to cover is the script/maldoc stuff. If you start with MA, then at somepoint you are going to need to learn RE.