Presentation at Hack.lu: Reversing the Broadcom NetExtreme's firmware
By guillaume » Sunday 21 November 2010, 04:28 - Conferences
I was giving a talk in October during last hack.lu session. The presentation focuses on the roadmap taken to reverse engineer the Broadcom Ethernet NetExtreme firmware family: building a firmware debugger, instrumentation tools, to finally develop a customized network card firmware.
NetExtreme family cards are the standard range of PCI Ethernet cards from Broadcom. Broadcom released part of their soft specifications (inner workings, memory mappings, device register definitions...). However those specifications are incomplete and the firmware is distributed as a binary blob.
Given publicly available documentation (specifications, Linux open-source driver) and free open-source tools, I have built a set of tools to instrument the network card firmware. Those tools provided me a way to debug in real-time the MIPS CPU of the network card, as well as doing some advanced instrumentation on the firmware code (execution flow tracing, memory-accesses logging...). I have also reverse engineered the format of the EEPROM where firmware code is kept and the bootstrap process of the device leading to firmware execution. This way it is possible to develop a custom firmware code, flash the device and get execution on the CPU of the network card.
The main interest is developing a rootkit which will be residing inside the network card. A network card rootkit offers some very interesting features:
- A very stealthy communication end-point over the Ethernet link. It can intercept and forge network frames without the operating system knowing about it.
- A physical system memory access using DMA over the PCI link, leading to OS corruption.
- No trace of the rootkit on the operating system, as it is being hidden inside the NIC.
The network card natively needs to perform DMA accesses, so that network frames can be exchanged between the driver and the device. From the firmware point of view, everything is operated using special dedicated device registers, some of them being non-documented. An attacker would then be able to communicate remotely with the rootkit in the network card and get access to the underlying operating system thanks to DMA.
Slides of my presentation are available here.
Here is a little demo of what can be done once you develop your own network card firmware:
Comments
Can you do similar reverses for common router firmwares?
I have a number of issues with this, firstly the code looks like Linux Kernel code. Let's see a demo where it's done on Windows.
Secondly most network cards aren't released with the EEPROM, a network card with an EEPROM can be programmed to to anything, that's the point of the EEPROM.
The code shown at the end of the demo is firmware code. It is meant to run on the device. This has absolutely nothing to do with the targeted operating system. And of course, code naming conventions has nothing to do with that either. Writing code with Windows conventions won't make your code more suitable to target Windows OS... This demo would work with Windows, or any OS, in the exact same way.
Secondly, assuming network cards do not have EEPROM is simply untrue. Only a few models for laptops with special low consumption feature do not come with an EEPROM. In fact, almost all models I've seen did have one.
And eventually yes, EEPROM can be reprogrammed...... wait...... so maybe that's why I reprogram it to inject and run my own firmware? I really hope you can see the point now.
Brilliant and frightening, I suspect that before too long there will be a range of attacks where all or part of the payload resides in flashable firmware.
The attack itself is not really new, similar attacks have been demonstrated in the past and pocs have shown how to infect PCI expansion blocks, bioses, or even worse, the memory controller hub.
If I understood coorectly:
1- Technically it is not trying to actively hide itself, so it would not be a real rootkit, it is pretty easily detectable by everyone who can dump the firmware memory. Unless you are planning to use a hypervisor... :)
2- As it is, the code is confined within the network card, and therefore has also limited malicious capabilities (eg it cannot bypass ssl encryption, it may try to access the physical memory and look for the cleartext data, but it would be quite complicated). To be really useful it should deliver a payload to the OS, and that payload would again be detectable
The very interesting part is the debugger, i think you did a great job, it would be a very good and useful idea to develop tools for auditing and analyzing network cards (and possibly any device's) firmware, hope you are planning to keep working on it and release such tools (or if they are already available i was dumb enough not to find it)
Thumbs up!
Dude, what about writing open source firmware for these cards.
really inspiring work! Thanks a bunch, and keep up the great work!
This is a bit disturbing if you consider something like an IPMI that was shipped with rogue firmware from the factory. That is just as possible as someone shipping malware on a photo frame, or USB drive.
On the other hand, I'd _dearly_ love to see the blobs get out of Linux. Hopefully this demonstrates how futile obfuscating firmware really is.
asdf@5: What do you mean, "has limited malicious capabilities"? On most systems (those without IOMMUs, and some with), any PCI card has unlimited access to all of physical memory. This includes the kernel page tables and the memory where SSL is keeping its keys.
You've got unlimited write access to /dev/mem/. How is this "limited malicious capabilities"?
This seems to be a somewhat similar problem to the "debug over firewire" issue that arose several years ago (in terms of impact, that is). http://www.mjmwired.net/kernel/Docu...
With compromised firmware on the NIC, a remote attacker would be able to read and modify any memory on the system. Basically a complete compromise.
On the other hand, this would be quite interesting as a debugging tool, for those who do not have a firewire interface on their device.
Either way, great research!
But what would happen, if we had reinstalled Operating System?
will attacker have an access to the device in this case?
I think this malware is futile. Vendors should sign their new NIC firmware releases with a secret crypto key or elliptic key, so that attackers cannot modify the existing firmware or create an unathorized new firmware due to the resulting incorrect checksum!
If the NIC card chip actually has a MIPS CPU core, that should be powerful enough to verify firmware integrity by calculating SHA-1/SHA-256 and maybe even send checksum requests to silicon-encoded, predefined IP addresses of the vendor to verify the veracity of the firmware being loaded.
Nice research !
I'm just not seeing lots of usecases. The major upside is that the malicious code is totaly hidden from the OS.
But since you first need to deliver this firmare and falsh the device... That requires an intrusion that you cannot hide from the OS, and you will still need to alter the OS to make sure it does not flash a fresh and clean firmware on a modprobe for example. So the uptime of the potential malicious paylod in the firmware is still limited, even cosidering you where able to flash and stay hidden...
But this is a hard kick in the microkernels's teeth since having userland-device-drivers don't help them on this one : PCI specs are the issue here ! And PCI is not about to die in the next few weeks ;)
@asdf: That's right, on the contrary of what I've read around about it, the rootkit is not undetectable. Basically I cannot prevent access reading to the device memory. Anyway I would say it is just very "well hidden", as it is not present on the hard drive of the infected machine.
@long: Reinstalling your OS will not remove the rootkit in any way, as it is stored in the NIC memory.
@Etienne: Of course the intrusion can leave traces, it is the role of the intruder to remove his traces from the OS after installing the rootkit. The device could also possibly be pre-rootkited. Talking about device reflashing, no driver actually does that (neither on Windows nor Linux/BSD). Once it is installed the rootkit can comfortably stay there for a long time.
Really cool research. Especially the debugging of the firmware.
Do you have any plans on releasing the source code for all this yet, as you have been talking about an open source replacement firmware?
Would a TPM detect this modified firmware? TPM measures the integrity of option ROMs, so I guess it would, but do you have any insight on that?
@Andy: Thanks! Yes, I will probably release my toolset first, the firmware development part is not terminated yet.
@Agadius: No, it won't detect it. The BIOS can only access the Expansion ROM, a specific zone in the EEPROM for which data access is proxied by the firmware itself. The rootkit is simply not stored in the Expansion ROM (which typically contains PXE boot code) if there is one. Even if it were, it could still hide itself as it has control over the data passed to the BIOS. The TPM will never have a chance to see the rootkit in memory.
I have two question:
-How can qubes OS help on it?
-Can IOMMU help in order to deal with this rootkit?
@Kevin:
That's a good question! I haven't talk about countermeasures in my presentation, because it was more about the reverse engineering process leading to firmware development than the rootkit itself. But yes, there do exist countermeasures and IOMMU is the first one of them.
In fact IOMMU is necessary to restrict DMA accesses from the device, but it is not sufficient alone. Given that the firmware is running way before the OS, it could potentially send DMA requests *before* the IOMMU gets activated. The solution is to couple it with a trusted-boot sequence. SRTM is theoretically insufficient because there is still the possibility to emit DMA writes from the NIC between the short time when the image integrity gets verified and when IOMMU is configured.
The proper way to deal with it is to use DRTM + IOMMU like Intel TXT / VT-d. TXT automatically enforce VT-d before doing integrity checks. The security check code can be executed in a DMA protected memory range (DPR). It can then safely reconfigure IOMMU before resuming the boot sequence.
I do not know Qubes very well, but I think using it with TXT would be a good help to protect against the rootkit DMA capabilities.
Here is a very interesting blog entry from Joanna Rutkowska related to your question :
http://theinvisiblethings.blogspot.com/2010/04/remotely-attacking-network-cards-or-why.html
I haven't reverse engineered the broadcom LAN cards, but I did make a set of drivers for Broadcom 802.11 wifi adapters and dell adapters w/ broadcom chipsets. The devices are much more capable than the standard drivers allow.
Magizian broadcom 802.11abgnm drivers (2 versions) boost range, speed, and add functionality/compatability i.e.. commview/airserv-ng compatability. .. and a few extra features.. they are available for free, http://tx0.org/18k , a few other http sites and you might might be lucky enough to find a seeded torrent.
Merry christmas, happy new year.
Mr. Guillaume Delugré, did you rewrite the whole firmware from scratch (in the demo video we see line 790 of the source code ...) on C?
@dimmo:
Yes, I have written a firmware from scratch in C language. Then I overwrite the default firmware with my own in the EEPROM.
Great work!
Do you plan to publish your code?
I research NIC BCM5761, and it would be *very* interesting to look at the sources.