Possible vulnerability in F5 BIG-IP LTM - Improper input validation of the HTTP version number of the HTTP reqest allows any payload size and conent to pass through

[Eitan Caspi <eitanc@xxxxxxxxx>]

Initial note: The vendor has graded this issue as a vulnerability graded as 
"High" in my email exchange with it, but eventually posted the issue as a "Know 
Issue", so some of this issue's characteristic that follows can be treated as 
initial ones, as I ask the IS community to look into this issue and give a 
"second opinion" about it. Thank you.


Suggested severity level: High (per the vendor's initial response)

Local / Remote activated: Remote

Exploit Code: No need for experiencing the issue, but may be needed to realize 
an exploit via this issue

Assumed cause for the issue: Improper input validation

Assumed types of Possible Risks/Abuses (all assumed as possible directions, 
since I did not have the time nor resources to look into them):
Denial of Service
Code Injection
(See more at "CWE-20: Improper Input Validation" - 
https://cwe.mitre.org/data/definitions/20.html)

Affected Software:
Vendor: F5 Networks
Product: BIG-IP
Module: LTM (Local Traffic Manager)
Versions: 11.6.0, 11.5.3, 11.5.2, 11.5.1, 11.5.0, 11.4.1, 11.4.0, 11.3.0, 
11.2.1, 11.2.0, 11.1.0, 11.0.0, 10.2.4, 10.2.3, 10.2.2, 10.2.1, 10.2.0, 10.1.0

Summary:
Input data accepted for the HTTP version number section of the HTTP request is 
not enforced to be in the correct format, hence any payload content/format and 
size is getting through without being blocked immediately by an error or 
security reply, and only when the underlying TCP timeout is reached – only then 
the base TCP connection is ending by the server side (i.e. no HTTP response is 
accepted from the remote server side)

Description, reproduction instructions and communication with the vendor (it is 
taken from the blog post I published about this issue, 
http://fudie.net/possible-vulnerability-in-f5-big-ip-ltm/, so it is somewhat 
written as a kind of a "story"):
"
About a year ago, while I was performing a web site penetration test for a 
customer, I run a manual fuzzing phase, where I like to “question” even the 
most basic networking and application conventions, and this time it paid off 
more than the usual…

The site was behind a “F5 Networks” BIG-IP device, running the modules of LTM 
(load-balancer, https://f5.com/products/modules/local-traffic-manager) and ASM 
(WAF (Web Application Firewall, 
https://f5.com/products/modules/application-security-manager)).

Generally, each HTTP request made by the client towards a web site (normally 
using a web browser) should have its first line in the following format:
<Method> <URL> <HTTP Version>

– The <Method> part is replaced with one of a fixed set of words that describe 
the action the client wish to perform on the target site (usually GET for 
viewing a web page) – The <URL> is the site address the client wishes to access 
– The <HTTP Version> must be in the format of HTTP/<version number> (usually 
0.9 or 1.0 or 1.1 which is the most common these days, e.g. HTTP/1.1)

For example:
GET http://www.somesite.com/ HTTP/1.1

I decided to attempt and disregard this format, so first I carefully tried 
messing with the HTTP version number – first I tried 0.1 (a non-existing HTTP 
version number) and sent something like:
GET http://somesite.com/ HTTP/0.1

To my surprise, instead of getting a reply of either an error message from the 
back-end web server or a security blocking message from ASM, for trying to be 
naughty – simply NOTHING came back, no reply arrived from the server side and 
the HTTP session ended only when its underlying TCP session timed out.

So, I dug dipper, to try and find what caused this phenomena, to see if there 
is some logic for this glitch – so… – I tried different numbers (e.g. 1000, 
0.5, -4, etc.) and the result was the same.
– I tried without any number (e.g. HTTP/ ) – the result was the same.
– I tried omitting the forward slash – the result was the same.
– I changed HTTP to be helloworld (GET http://somesite.com/ helloworld) – the 
result was still the same.

So, it looked like there is no format verification and enforcement being made 
at the BIG-IP end regarding the HTTP version part of the request, a situation 
that looked risky to me, as the content/payload being pushed from the client to 
the server is not sanitized properly, possibly being accumulated at some 
memory, either on any relevant F5 BIG-IP module or at the back-end application 
server, which may lead to server resources being exhausted or possibly 
overflowing the memory, which can be much more dangerous, as it may allow the 
attacker to run there his/her own malicious code at the remote server side.

I tried to run some basic low-power DoS (Denial-of-Service) attacks using this 
issue, to see at what level, if at all, this issue can harm the target system – 
but since I didn’t have the sufficient resources to perform a decent large 
scale DoS attack, I wasn’t able to spot anything more than a minor traffic 
delay, but not something that I can be sure that this issue was the cause of it.
So, I had to stop my research at this point and turn to the vendor, “F5 
Networks”.



First of all, I found the web page in which F5 states about its security 
vulnerability response policy (“sol4602: Overview of the F5 security 
vulnerability response 
policy“,https://support.f5.com/kb/en-us/solutions/public/4000/600/sol4602.html).
 Unfortunately this page did not include any public key (using PGP) or S/MIME 
certificate to encrypt the email exchange between me and F5 nor any SSL based 
web form.
So, since I wanted to report my finding securely, I sent on the 8-Feb-2015 the 
following email to the email address mentioned at the above web page as the one 
for reporting security issues – security-reporting@xxxxxx. I signed this email 
with my S/MIME certificate, so F5 will have a way to securely reply to me.
1. I wrote:
”
Hello,

I wish to report something to F5, but at 
https://support.f5.com/kb/en-us/solutions/public/4000/600/sol4602.html?sr=12234302
 there is no mention of any way to security encrypt the email sent to F5, 
either PGP or s/mime – do you have something in place for this goal? (s/mime is 
preferred)

Thanks.
”

2. The next day, 9-Feb-2015, I was replied by F5 Support Reply 
<SupportOne@xxxxxx>:
”
Hello Eitan,

I have forwarded your request to our IT department, as I spoke with one of the 
security people and he does know what you are asking for, someone will be back 
in contact with you soon in regards to your request.

Respectfully,

<employee-name> | Technical Support Coordinator II ”

3. I promptly replied on the same day:
”
Cool, thanks.

I will be waiting for your move.
”

4. As no one approached me by the 14-Feb-2015, I sent the following email to 
the support email address I was replied from initially:
”
Hello F5,

Any news on this?
”

5. I was replied on the same day with:
”
Hello, Eitan:

I spoke with our Duty Manager about your security concerns.  He followed up 
with our IT Security Department.  I was told that they do not have an answer at 
this moment, but they are working on the problem.

Meanwhile, I am leaving a note for <employee-name>, the woman with whom you 
communicated last week.  She will be in tomorrow, and can get in touch with the 
team originally tasked with this.

I apologize for the length of time it is taking to resolve this, but I assure 
you that we are taking measures.

Regards,

<employee-name>

F5 Networks Support | www.f5.com
”

As no one replied to me by the 21-Feb-2015 I sent the following email (which 
included all of the above correspondence) to Mr. Manny Rivelo, who at the time 
was the company’s senior vice president in charge of security (and now he is 
the former-CEO, 
https://f5.com/about-us/news/press-releases/f5-networks-announces-appointment-of-long-time-f5-executive-john-mcadam-as-president-and-ceo),
 and I was in touch with him a few times during the time I worked at F5:
”
Guys, are you serious?! How long does it take?

You don’t treat security seriously.
”

On the same day he replied from his phone:
”
Eitan, I will reach out to our CIO to see what the issue is.
”

I replied on the same day:
”
Cool, it will help my will to securely hand F5 via email a possible security 
issue and also for others in the future.
Consider also adding a secured web form for this purpose, as a parallel and 
quick way to submit security issues.

In security response time is crucial.
”

Two days later, on the 23-Feb-2015 I was replied by M., a senior security 
supporter, who from now on will be my point of contact, and this was the first 
time I received an official support case number, to track this case:
”
Eitan,

First, apologies for the delay in responding to you.  I wasn’t involved, but 
I’ve been told there was some confusion with our first tier of support which 
caused a delay in getting this into the proper channels.

I’ll email you directly in a moment.  I don’t have S/MIME, but we do have two 
options for secure communication.  The first, and easiest, is <a service F5 
uses. I will omit it as it is not relevant here and it is a quite long text. 
Eitan>

The second method we can use is PGP.  My public key is listed on MIT’s 
keyserver: <link>

I’ll email you directly twice – once with each method – to make things easier 
for you.  Then you can respond using your preferred method.

Thank you.

-M.
”

I replied to M. on the same day:
”
Hello M.,

Thank you for jumping in to save the day… 😉

Got your key and added to my keyring (btw, consider giving it an expiration 
date).
I don’t publish my PK on the net, but I added it here for you.

I will re-verify the issue, write my report about it and send it your way in 
the coming days, now that I am sure we have a secure (relatively, these days…) 
communication channel.

Will be in touch soon.

Cheers!
”

I sent M. the report using an encrypted email and on the 2-March-2015 he 
emailed me:
”
Eitan,
I received and successfully decrypted your email from Saturday.  I’ll attempt 
to produce the issue in our lab and I will let you know the status.
Thank you
”

On 8-May-2015 M. wrote to me:
”
Eitan,

We worked on this with our product development group and we’ve opened ID518020 
for this issue.  We are classifying it as a Vulnerability, with a Severity of 
High under our policy – covered in SOL4602.  We will be fixing this in our next 
major release, 12.0.0, due out later this year.  We’re also planning on pulling 
the fix back to future Hotfix Rollups on our current supported releases.  That 
work is already underway.

From our testing, overall it appears that the behavior is basically similar to 
what we do if we get a connection that doesn’t send a complete request; we’ll 
eventually disconnect the client via the TCP profile timeout.  The reason these 
requests aren’t rejected outright is that the profile currently looks for a 
valid HTTP/1.0 or 1.1 request format – basically <method> <uri> HTTP1.[0|1] – 
but if it isn’t one of those it presumes it must be HTTP/0.9, which was much 
more forgiving with respect to the headers.  (HTTP/2 is experimental in the 
latest 11.6.0 releases, but that’s a different beast entirely.)  So the current 
profile errs on the side of trying to service the request and falls back to 
0.9; a server which strictly enforces 1.0 or 1.1 is likely to reject them 
outright.

We try to process this HTTP/0.9 request and look for a response from the 
server, but things break down when we try to proxy the bad request.  The short 
version is that we send a request to a pool member, but it sees it as HTTP/1.0 
and since it isn’t a complete 1.0 header it is waiting for more.  We end up 
sitting in the middle until our timer fires and we close the connection or one 
of the endpoints times out and closes it, whichever comes first.  So this all 
stems from trying to be too accommodating with support for HTTP/0.9 and the fix 
is to get stricter about what we’ll accept.

The good news is that we’ve conducted volumetric automated testing and we 
haven’t been able to produce any noticeable issue with the BIG-IP; it has 
absorbed the queries as intended.  In theory a very high number of requests, 
from a larger number of attacking systems, could create a DoS situation, but 
the behavior appears to be the same as other known DoS methods in being related 
to the TCP profile timeout.  Even with 65,000 simultaneous connections (using 
all available source ports, in other words) from one host it barely moved the 
needle on the BIG-IPs resource utilization, so it would need to be a DDoS type 
of attack from a large number of attack hosts.  The actual cost per connection 
to the BIG-IP is negligible, it isn’t consuming much in the way of resources.

Overall we do see a few ways we could enhance the BIG-IP’s internal filters to 
more strictly enforce the request format and reject this kind of exploit in 
general more readily, and we’re looking into what improvements can be made in 
future releases.   Based on our testing, this attack has about the same impact 
on the BIG-IP as opening the TCP connection and sending nothing, until the TCP 
profile closes the connection.  It isn’t as resource intensive as Slowloris for 
example, though that’s obviously true on the client side as well, making this 
attack lower cost.  We tested a number of variations with different types of 
malformed headers, looking for any different characteristics.

We do appreciate your bringing this to our attention as it has given us a few 
ideas on ways we can further improve the product.  I’ll be working with our 
AskF5 team to prepare a Solution article for our site, and we can coordinate 
the publication with your timing as well.

I hope this explanation addresses the concerns you’ve raised.  Do you have any 
questions I might address?

Thank you.
”

Following this email we had some more emails exchange regarding various 
possible risk and mitigations about this issue, and I was asked to supply the 
title I wished to be presented in the acknowledgment part of the vulnerability 
support post.

At the 27-May-2015 I received an email from M. that eventually they published 
the issue at the support site, titled “SOL16672: An improperly formatted HTTP 
request-line may cause connections to hang and eventually 
timeout“,https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16672.html.

According to this KB this issue is affecting versions ranging from version 
10.1.0 (from December 2009) until a very recent version – 11.6.0 (August 2015) 
(Click the “Show Version” part on the upper right side of the KB post).

To my surprise the issue was not marked as a vulnerability but as a “Known 
Issue”, although during my discussions with “F5 Networks” they classified it as 
a “High” grade vulnerability (see above), and no one updated me, before 
publishing the support post – that they decided to demote it from a 
vulnerability.
Even the “Acknowledgments” section was deviated from the normal text pattern 
they use for vulnerabilities posts (see here an example here - 
https://support.f5.com/kb/en-us/solutions/public/16000/700/sol16728.html)

I asked M. why, after they admitted it is a “High” grade vulnerability by their 
own policy – they eventually  played it down to a “Known Issue”?

He replied:
”
Eitan,

There was a lot of discussion about this internally, but in the end the 
developers and management decided this is not a vulnerability in BIG-IP, but a 
defect.  The ID was re-categorized as Defect and is being addressed as such.  
Hence there will be no CVE requested and the SOL format used was the Known 
Issue template used for product defects.

Through the testing that was done they were never able to produce a real 
DoS/DDoS effect on the BIG-IP and they didn’t feel that it rose to the level to 
be classified as a vulnerability.  But they did feel that it was not the proper 
handling of these requests, so it is considered a defect to be resolved in 
upcoming HF releases.  It was a very involved debate, with a lot of arguments 
made for both sides.  In the end the ‘Defect’ arguments carried the day.

The central point was really trying to rectify the test results, wherein little 
impact was observed on the BIG-IP, with calling it a Vulnerability and then 
trying to explain the minimal impact in a SOL.  There was concern that labeling 
it as such would cause undue concern for customers in light of the negligible 
impact observed, as there is something of a strong reaction to any 
Vulnerability SOL independent of the content.  It was agreed however that 
customers should be made aware of the issue in any case, and the Known Issue 
SOL was expedited for that purpose.

I had initially argued to handle it as a Vulnerability, but in the end I agreed 
with the consensus to handle it as a Defect after all of the arguments had been 
presented.
”

I publish this post because I guess most people are not aware of this issue as 
it was not flagged as a vulnerability hence folks who collect and react to 
vulnerabilities report could not possibly know about it.

I hope that with this post more “F5 Networks” customers will be aware of this 
issue and patch their systems with the fixes mentioned at the KB post (or 
instead use the suggested mitigation steps offered there as well); and that the 
security researchers community will learn about this issue, thus researchers 
with better tools, knowledge and experience than I have – will look deeper into 
this issue and give this issue a “second opinion”, whether it can be exploited 
as a vulnerability or not.
"

Direct binary based resolution: Download and upgrade to the latest main version 
of the LTM module, begging from version 12.0.0 or install a hotfix for matching 
earlier versions, per the vendor's support article of "sol16672: An improperly 
formatted HTTP request-line may cause connections to hang and eventually 
timeout" at 
https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16672.html

Workaround (in case you cannot apply the above binary code or do not wish to do 
so):
1.    Create an iRule (an internal BIG-IP script) to enforce correct HTTP 
version input format validation 2.    In addition it is possible to lower the 
TCP timeout value Exact details about these workarounds can be found at the 
vendor's support article of "sol16672: An improperly formatted HTTP 
request-line may cause connections to hang and eventually timeout" at 
https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16672.html

Credit: Eitan Caspi, Israel

Past security advisories:

1. CVE-2002-0049 - Exchange 2000 System Attendant Incorrectly Sets Remote 
Registry Permissions 
http://www.microsoft.com/technet/security/bulletin/MS02-003.mspx
http://support.microsoft.com/kb/315085/en-us
http://online.securityfocus.com/bid/4053
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2002-0049

2. CVE-2002-1932 - Defined Actions for Administrative Alerts Do Not Occur When 
the Security Log Is Full
http://support.microsoft.com/?kbid=329350
http://online.securityfocus.com/bid/5972
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2002-1932

3. User downgraded from Administrator to User retains the ability to list other 
user's running tasks
http://www.securityfocus.com/archive/1/301624
http://online.securityfocus.com/bid/6280

4. "Compaq Web Agent" management session can be re-used without the need to 
perform authentication
http://online.securityfocus.com/archive/1/309442
http://online.securityfocus.com/bid/6736

5. Windows XP "welcome screen" exposes the names of all the members of the 
local administrators group
http://www.securityfocus.com/archive/1/314361
http://www.securityfocus.com/bid/7046

6. Symantec Antivirus client locally created scheduled scan is not running if 
the local console is logged off
http://www.securityfocus.com/archive/1/393800

7. CVE-2006-2612 - Novell Client login form enables reading and writing from 
and to the clipboard of the logged-in user 
http://www.securityfocus.com/archive/1/archive/1/434704/100/0/
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-2612

8. CVE-2006-4886 - McAfee VirusScan Enterprise - disabling the client side 
"On-Access Scan"
http://www.securityfocus.com/archive/1/archive/1/446220/100/0/
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-4886

9. CVE-2007-0833 - VMware workstation guest isolation weaknesses (clipboard 
transfer) http://www.securityfocus.com/archive/1/459140/30/90/
http://www.securityfocus.com/bid/22413
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-0833

10. CVE-2007-1056 - VMware Workstation multiple denial of service and isolation 
manipulation vulnerabilities 
http://www.securityfocus.com/archive/1/460664/30/60/
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-1056

11. "run as" local denial-of-service enables administrative account processes 
to be killed www.securityfocus.com/archive/1/472216/100/0/

Email: eitancaspi (at) yahoo (dot) com or via the "Contact" form at the 
following blogs

LinkedIn Profie: https://www.linkedin.com/in/eitancaspi

Information Security blogs:
FUD for thought (English) - http://fudie.net Not Safe/Sure (Hebrew) - 
http://security.caspi.org.il

Articles: You can find several IT, business and security articles I wrote some 
time ago at 
http://www.themarker.com/misc/search-results?searchType=textSearch&text=eitan+caspi&simpleSearch=simpleSearch

"Technology is like sex. No hands on - No fun." (Eitan Caspi)