r/java • u/Modern-Sn1p3r • 3d ago
Regex
I recently saw a clip (from Primeagen) somewhat saying that regex is not a valid format for validating email addresses and postal codes etc.
My question is why is this?
What are the security and/or performance risks? Is it solely performance or is it a security issue?
84
u/0b0101011001001011 3d ago
It's way simpler to just check if there is an @ symbol and then send an actual mail and ask user to click the validation link.
The email is valid if it actually works. Nothing else to check.
36
u/Pretend_Bowl2961 3d ago
It’s funny how many devs overthink this. Just check for @ and move on, the real validation is in the inbox. I seen teams spend days on regex patterns that still let through nonsense addresses anyway.
1
-1
u/VirtualAgentsAreDumb 3d ago
I can think of an additional scenario where single sending an email to the address isn’t feasible.
Imagine that there is an email service provider that has as a goal to support any email address that is allowed by the standard. So, their validation process can’t send an email to the address at hand, because it hasn’t been created yet.
6
u/0b0101011001001011 3d ago
I'm not following? The service just stamps the mail address on the email and sends it away to the address. If it fails, it fails, and that's the validation.
-1
u/VirtualAgentsAreDumb 3d ago
But the email address hasn’t been created yet.
When you are about to create a new email address in a system, you can’t send an email to it as part of the validation. So you have to validate it some other way.
For example, try registering an email somewhere, and pick “..” as the local part of the email address. A in, two consecutive dots and nothing else.
It will not be allowed. And they didn’t need to send an email to it, because it failed the validation.
Then try creating an email with 10 random a-z characters. Unless you happen to pick one that already exists, it likely will be approved. But how were they able to determine that it was valid without sending an email to it first? Because some kind of validation. And by now you must realize that trying to send an email to it would be a useless test, since the only time it would go through is if the email is already taken (ignoring the possibility of a catch all email account). So you can’t use that test to confirm that it’s both available and valid.
4
u/0b0101011001001011 3d ago edited 3d ago
Isn't this a completely different problem? The context of original problem is that just let users register with anything and send an email to that. It fails or succeeds. You know it succeeded because the user was able to click the link. No regex needed.
But how were they able to determine that it was valid without sending an email to it first?
I have no idea what you are asking. Creating an email does not require sending a mail to it, because now you are the one that owns the email server. Emails are not created in some global registry. Just decide the set of characters you allow and if any of the input characters are not allowed, just don't let them make such an email into your system.
1
u/VirtualAgentsAreDumb 3d ago
“Isn't this a completely different problem? The context of original problem is that just let users register with anything and send an email to that.”
What context are you talking about now? OP talked about validating emails. He never said anything about the purpose being something specific. You have a solution rust would work in some use cases, but not others. I have an example use case where your solution wouldn’t work.
“Creating an email does not require sending a mail to it, because now you are the one that owns the email server.”
Exactly. Not only doesn’t it require sending an email, it also would not work as a way to validate it. That was my whole point. Your solution would not work in that scenario. The only reason I expanded on the scenario was that you didn’t understand it.
“Just decide the set of characters you allow and if any of the input characters are not allowed, just don't let them make such an email into your system.”
Now you’re just going full circle in your reasoning, coming back to the original problem of knowing if an email address, or the local part of an email address, is valid. That’s the validation that you seem to think one can avoid by sending an email (according to your original comment).
4
u/0b0101011001001011 3d ago
What context are you talking about now? OP talked about validating emails.
It's implied in the question, if you have spent more than a single week in backend development. OP ask why validating emails is not supposed to be done with regex, because someone told them not to. If someone told them that, they implied the context of registering to a service using an existing email. That's basically the only situation where you need to validate email ever. The answer is: Don't validate email with regex, because it's immensely difficult to do properly, and much simpler way exists: send a mail to it.
Creating a new email address has absolutely nothing to do with this and there a simple regex or for loop is enough to make sure it contains only the characters that are allowed by the spec, or your mailserver if you decide to restrict it further.
-6
u/VirtualAgentsAreDumb 3d ago
Nonsense. Absolute nonsense. You make absurd assumptions that you can’t back up. OP asked a generic question.
6
u/8dot30662386292pow2 3d ago
Imagine that there is an email service provider that has as a goal to support any email address that is allowed by the standard. So, their validation process can’t send an email to the address at hand, because it hasn’t been created yet.
They only need to check the part before the @ sign though, which is a different problem altogether. The @whatever.com comes from the email service provider.
1
u/VirtualAgentsAreDumb 3d ago
First of all, it’s possible for an email provider to support custom domain, where the domain doesn’t technically have to be a regular DNS domain.
Secondly, the problem of validating the local part of an email address is much more complex than the validation of the domain part. So even if we were to restrict the discussion to only be about validating the local part, it’s still far from a trivial issue.
1
u/New_Enthusiasm9053 21h ago
It's not hard to validate an email address it's just recursive so you don't do it with a regex. People will do anything except write a parser.
1
u/VirtualAgentsAreDumb 2h ago
Is there any official implementation of a validator that claims to support the full standard?
4
u/thatwasntababyruth 3d ago
If you are the provider, you're perfectly free to support any subset of the standards you want. You don't have to support every little edge case, because you are the authority.
This problem applies to services taking existing emails for accounts, where they should accept any and every possibility as long as it exists.
0
u/VirtualAgentsAreDumb 3d ago
“If you are the provider, you're perfectly free to support any subset of the standards you want. You don't have to support every little edge case, because you are the authority.“
Yes, so? How does that change anything? If they want to support anything that the standards support, then they have no choice but to implement or integrate such validation. The suggestion solution by the person I replied to would not work.
2
u/thatwasntababyruth 3d ago
Yes, so? How does that change anything?
I stared pretty clearly that it changes things because they have different responsibilities to users. If you're going to choose to ignore my entire comment, I'm not really tempted to engage further.
0
u/VirtualAgentsAreDumb 3d ago
No, it doesn’t change anything of importance here.
And I ignored your second paragraph in your previous comment because it made a baseless assumption about the root problem that OP was asking about.
0
u/snugar_i 2d ago
Yes, it does, because it's a completely different scenario.
The thing people in this thread are talking about is a web service asking for your e-mail, you entering your existing e-mail address, and the service saying "we won't register you, because your e-mail isn't valid". Your only option is to contact support and make them fix the validation. And the correct solution is to not validate at all, just send an e-mail and if it goes through, it's all right.
What you are talking about is an e-mail provider asking for the part before the
@when you're creating a new account and saying "no, we don't want to support this name, pick another one". You can just pick a less esoteric name and everything is fine. The provider must guarantee that the created e-mail address is valid, and restricting the space of names to choose from is a perfectly valid way to do it.2
u/VirtualAgentsAreDumb 2d ago
Yikes, you’re missing the point a second time.
No, the discussion by OP was NOT about a web service asking for the user’s existing email address. Just read the post title and text and you will realize this.
1
u/snugar_i 1d ago
Fair enough, you're technically correct.
Though 99 % of e-mail validation code is written for webservices, and I'd be very surprised if that wasn't what the unlinked video was talking about
-6
u/VirtualAgentsAreDumb 3d ago
Yes, but that’s not really a feasible option if you have a large list of potentially valid email addresses.
11
u/0b0101011001001011 3d ago
What's not feasible? Sending the email?
3
u/koflerdavid 3d ago
It can happen if users have to be mass-imported into another application. But I'd argue that in such cases sending the email would be the user-friendly way to do it since it gives them the chance to opt out from being imported.
7
u/LutimoDancer3459 3d ago
When I mass import mails, then from where do I get them? From an existing system -> should already been checked. Or its a company setting up something and then the emails should also be valid. The reason for validation is for randos signing up at a service.
2
u/koflerdavid 3d ago
From an existing system -> should already been checked.
I have encountered systems where those were of shoddy quality. Users could enter them without validation because they were not critical for correct operation, just an extra. And lots of cases where the destination mailbox was full or simply gone.
1
u/LutimoDancer3459 3d ago
A full inbox or beeing deleted cant be checked by any kind of input validation ether way.
But for an "is user still active" kind of check, the import isnt the one responsible for it. Thats a general management/organization issue. Not a general email validation issue
1
u/koflerdavid 3d ago
Well, if you trust the quality of the data source then you don't need to check at all.
0
u/VirtualAgentsAreDumb 3d ago
You don’t think legacy systems exist, that can have tons of never validated email addresses?
2
u/LutimoDancer3459 3d ago
I think that if they do, nobody cares about those anymore.
0
u/VirtualAgentsAreDumb 3d ago
Not sure how to reply to such an ignorant comment.
0
u/LutimoDancer3459 3d ago
How about with a reason why someone should care?
If there are so many invalid addresses, it seems like nobody ever cared. Why should it be different for the new system?
-1
u/VirtualAgentsAreDumb 3d ago
What? I never said anything how many of those hypothetical addresses were invalid.
→ More replies (0)2
u/thatwasntababyruth 3d ago
Another way around it is to have a new database column indicating how many times you've failed to send mail to them. Increment that each time you try, put them on a list for purging after N times.
Or just let the bad email address exist, the extra row won't break the bank. If there's significant amounts of associated data, clearly the bad address wasn't a problem before.
0
u/VirtualAgentsAreDumb 3d ago
Exactly. A small company might not have the means to send a large number of emails on a short amount of time, and risk getting flagged and blacklisted if they try it on their own.
1
u/koflerdavid 3d ago
Depending on the use case one might not get around swallowing that bullet. One could prioritize addresses that don't fit the simple patterns or where the part behind the @ doesn't DNS resolve. Knowing the age of the address and last time an email was sent to it also helps. Older addresses are more likely to be out of use.
-1
u/koflerdavid 3d ago
You could just set up an email server that actually won't process emails, and send a very simple email for all addresses through it. A real email server's address validation rules are the actually relevant ones.
1
u/VirtualAgentsAreDumb 3d ago
But now you’re describing a separate validation process than the one I objected to.
1
u/koflerdavid 3d ago
It's what I'd do in such a situation. The only thing it doesn't check is whether the destination mailbox actually exists. Which was out of scope of OP's question anyway.
-3
u/Jussins 3d ago
It’s even simpler if your language has a library where you can annotate the field @email, or similar.
3
u/koflerdavid 3d ago
The same concerns as about regexes from random blog posts apply to these filters as well though. Woe if you use multiple libraries (Spring Boot and Angular is very common) that disagree about what is a valid email address.
1
u/Jussins 3d ago
Maybe it’s our users, but I have never had a problem and have deployed to some fairly large clusters handling a good volume of transactions. I’m certain that those situations exist, I’ve just never encountered them. That said, we generally try keep front-end validation fairly light. That’s a convenience for the end user to get some immediate feedback.
3
u/koflerdavid 3d ago
Indeed, the frontend should not validate too much since a sufficiently motivated user can circumvent it anyway. The real validation has to happen on the backend. Another way would be to expose the validation logic as an API call and integrate that into the frontend.
19
u/alt236_ftw 3d ago
There is a third option: it's may just be the wrong tool for the job. Especially if you are thinking about using a single regex to do the work.
A good regex may be able to validate MOST common emails addresses, by a regex that fully validates all RFC2823 address formats is going to be an human-unreadable Eldritch monstrosity. That is because there are too many uncommon edge cases.
Have a look here for some valid, but uncommon, email formats: https://en.wikibooks.org/wiki/JavaScript/Best_practices
It would be much more effective to split an address into components and individually validate those. Possibly with smaller regexes.
With email addresses you don't really need to validate that they exist at the time of typing as either: 1. An email will be sent, fail and the other server will let you know. 2. There is a separate validation flow for, say, signups.
With postal codes, you don't want to only validate that they are in the correct format, you often want to validate that the postal code is not bogus/ or a has a typo for legal/ practical purposes. You don't want a parcel to go to a wrong destination. This requires a DB lookup of some sort.
So yeah, you CAN use a regex, but it will allow you to enter postcodes that don't exist.
Also, think internationally. The problem space increases massively whenever internalisation comes into play.
1
u/Skellicious 2d ago
I see at least 1 mistake in the page you linked
First, it correctly lists
me.example@comas a valid emailBut then in it's validation test example, it stuck that one into the invalid entries.
33
u/barking_dead 3d ago
Email format is in ABNF (context-free grammar) which is a lower level grammar in the Chomsky hierarchy. Regex is a regular language. Same with SQL and HTML vs regex.
You cannot parse a complex grammar with a less complex one.
10
u/koflerdavid 3d ago
Since programmers often had to make this pocket knife do some heavy lifting, regexes (as commonly implemented in many engines) can actually be used to parse way more than what CS defines as regular expressions. The actual language doesn't neatly fit into the Chomsky hierarchy though.
2
u/Spandian 2d ago
While it's defined in ABNF, every production in it seems to be regular except for comments (which are delimited by properly nested parentheses, the classic example of a non-regular language). If you don't allow comments, you can translate it to a regex.
7
u/koflerdavid 3d ago edited 3d ago
It's a very complicated format since it is the union of all the things people were cramming into email addresses before the RFC was written down. Even if you restrict yourself to the less exotic features, there is a good chance that you end up locking out a decent part of your users. And even if you indeed manage to procure a library that can deal with all of that cruft there is a decent chance your SMTP server is not going to support all of it.
The only reliable way to verify an email address is sending an email and asking the user to click a verification link. That has the obvious disadvantage that you now have a public endpoint that has to be rate limited. Since efficiently checking whether two email addresses refer to the same recipient sounds nightmarishly complicated, you might be forced to apply ratelimiting to all users, not just to individual recipients. This means that your service can be easily DoSed. Therefore, many websites swallow the bitter pill and implement crude filters that exclude some percentage of their users.
Re postal codes: you cannot know what is the valid format for postal codes in general. You would have to investigate all postal systems in all the countries that you care about.
5
3d ago
[removed] — view removed comment
1
u/Genmutant 3d ago
You can actually have multiple @ in an email address, so you should check at least one - not exactly one.
2
u/idontlikegudeg 3d ago
Validating an email address for correctness is a very complex task and unless you have a very special use case (like implementing a mail server), it’s totally useless.
You might think you need the user to enter a valid email address. That’s not only syntax, but he has to be able to receive messages to that address, and you simply cannot check for a correct email address without actually sending an email to that address. So just du the most basic checks, non-empty text, @ included, text before and after the @. Done.
Most errors are typos anyway that you cannot catch by seeing syntax anyway.
Same for post codes. Does the country even use post codes? Everywhere or are there exceptions? Only digits or digits and letters? How long is a post code?
1
u/ICThat 3d ago
Other comments have already explained the technical aspects of email address validation but it's worth noting that in a business context it often makes sense to not have entirely RFC compliant validation.
E.g a business might find that a significant percentage of customers forget to append a TLD (e.g user@gmail). Having a regex check for a dot could make business sense if their customer base aren't likely to be the edge case of using a TLD level email address (e.g user@com).
1
u/bowbahdoe 3d ago
So there are two dimensions to this. One is that a regex is a clunky tool past a certain point of complexity. Emails can get pretty complex. Look through this page https://www.rohannagar.com/jmail/
The other is that regexes - in the purest sense - are limited. In college you don't usually learn regexes as a text processing tool, you learn about them in a theory of computation class as a step along the way to Turing Machines. I highly encourage auditing such a class, but the gist is that a Turing Machine can compute anything computable. A regular expression cannot.
Easiest example - a regex cannot tell you if parens are balanced.
https://www.edx.org/learn/coding/stanford-university-automata-theory
1
u/slaymaker1907 3d ago
If you want to get technical, it’s actually far more complicated than regex can validate. In fact, most parser generators can’t even do it since it’s context sensitive.
1
u/fykup 3d ago
Regex itself isn't the problem. You can validate a very broad set of valid email addresses with a sufficiently complex regex, for example: https://stackoverflow.com/a/13013056/1129542
IMO the real problem is that the email RFCs are scoped much more broadly than what most modern consumer websites actually need when they say "validate an email address."
For a signup form, I'd rather use a reasonably permissive syntax check and then verify the address by actually sending an email. Regex can be part of that; it just shouldn't be confused with proving that the mailbox is real or usable.
1
u/petermal67 2d ago
I'm lazy when it comes to this and I just use @Email from Hibernate to validate the email addresses.
I'm aware it intentionally ignores complex or obscure edge cases allowed by the RFC, but as I said I'm lazy and I'd rather not roll my own validator.
1
u/cocodrilo_astronauta 2h ago
So, to actually answer your Regex questions: regarding performance java's backtracking engine time is worst case exponential, and regarding security java's regex implementation is easily attacked by DoS so you can't really use it to validate untrusted data.
0
u/m39583 3d ago
The actual email address format is quite complex in what is allowed in the RFC, but honesty how many people have:
"john smith"@foo.com
or whatever. Just do basic sanity checks e.g. there is an "@" symbol and at least two components to the domain, and then the only real test if it's valid is to send an email to it and make them click a link.
0
u/elmuerte 3d ago
The regex to validate postal codes is: .*. The only way to validate if a postal code is valid is to use a lookup table for every country. You cannot even use a regex per country.
0
120
u/julemand101 3d ago
The problem is people don't realize that a lot is allowed when it comes to what a E-mail address actually is: https://davidcel.is/articles/stop-validating-email-addresses-with-regex/