I’m working on customizing the LLVM x86 backend and need to transform specific indirect jump patterns. My goal is to change jmp *(edx) instructions into a two-step sequence like mov *(edx),ecx; jmp *ecx during code generation.
First, I want to add logging functionality so I can see when the compiler encounters these jmp *(edx) patterns. I’m thinking of adding debug output statements to track these occurrences.
After getting the detection working, I plan to implement the actual instruction replacement logic.
I’ve been researching this and it seems like modifying the x86 assembly printer in LLVM’s backend might be the right approach, but I’m not certain about the exact steps. Can someone point me toward the right components to modify or suggest some documentation?
This is part of a larger project involving indirect jumps and stack operations, but I’m starting simple to learn how the backend works first.
X86ExpandPseudo pass is your best entry point here. I hit similar requirements when implementing custom instruction sequences - intercepting at the pseudo-instruction expansion phase gives you clean control over the generated machine code. Check out X86ExpandPseudo.cpp in lib/Target/X86/. This pass runs after instruction selection but before register allocation, so it’s perfect for your two-instruction transformation. Look at how existing pseudo instructions expand into multiple real instructions. For logging, just add debug output using LLVM_DEBUG macros in the expansion logic - you’ll see exactly when your patterns match. The trick is identifying the right machine instruction opcode for your indirect jump pattern before you implement the replacement logic.
check out x86instrinfo.cpp and think about writing a custom machinefunctionpass. max is right - the assembly printer won’t work for this. look at how other backends in lib/target/x86/ handle smilar transforms. llvm’s pass writting docs are pretty helpful too.
cool project! why not try machine instruction passes instead of the assembly printer? the printer’s probably too late in the pipeline for clean transforms. what x86 target are you using - specific architecture or just general x86? what’s the bigger picture here?