Current state of the art
Factorization of integer in polynomial time is still to date an unresolved problem. Cryptographic algorithms like RSA, Rabin all rely upon the difficulty of integer factorization problem. Even to date factoring large integers with very fast computers require a lot of computing time. There are some efficient pseudo polynomial time algorithms known for the factoring problem.
Difficulty and complexity
If a large, b-bit number is the product of two primes that are roughly the same size, then no algorithm is published that can factor in polynomial time. That means there is no widely known algorithm that can factor it in time O(bk) for any constant k. In other words, there are algorithms which are super-polynomial but sub- exponential. In particular, the best published asymptotic running time is for the general number field sieve (GNFS) algorithm, which, for a b-bit number n, is:
For an ordinary computer, GNFS is the best published algorithm for large n (more than about 100 digits). For a quantum computer, however, Peter Shor discovered an algorithm in 1994 that solves it in polynomial time. This will have significant implications for cryptography if a large quantum computer is ever built. Shor's algorithm takes only O(b3) time and O(b) space on b-bit number inputs. In 2001, the first 7-qubit quantum computer became the first to run Shor's algorithm. It factored the number 15.
It is not known exactly which complexity classes contain the integer factorization problem. The decision-problem form of it ("does N have a factor less than M?") is known to be in both NP and co-NP. This is because both YES and NO answers can be trivially verified given the prime factors (whose correctness can be verified using the AKS primality test). It is known to be in BQP because of Shor's algorithm. It is suspected to be outside of all three of the complexity classes P, NP-Complete, and co-NP-Complete. If it could be proved that it is in either NP-Complete or co-NP-Complete, that would imply NP = co-NP. That would be a very surprising result, and therefore integer factorization is widely suspected to be outside both of those classes. Many people have tried to find classical polynomial-time algorithms for it and failed, and therefore it is widely suspected to be outside P.
Interestingly, the decision problem "is N a composite number?" (or equivalently: "is N a prime number?") appears to be much easier than the problem of actually finding the factors of N. Specifically, the former can be solved in polynomial time (in the number n of digits of N) with the AKS primality test. In addition, there are a number of probabilistic algorithms that can test primality very quickly if one is willing to accept the small possibility of error. The easiness of primality testing is a crucial part of the RSA algorithm, as it is necessary to find large prime numbers to start with.