Google App Engine's `DistributionNotFound: google-cloud-ndb`

When following Google’s migration guide for GAE 2.7 to GAE 3.7, they mention to start using their new ndb client google-cloud-ndb. This is straighforward until we ran into the following problem:

Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 311, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/python27/python27_lib/versions/third_party/setuptools-0.6c11/pkg_resources.py", line 311, in get_distribution
    if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/python27/python27_lib/versions/third_party/setuptools-0.6c11/pkg_resources.py", line 197, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/python27/python27_lib/versions/third_party/setuptools-0.6c11/pkg_resources.py", line 666, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/python27/python27_lib/versions/third_party/setuptools-0.6c11/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
DistributionNotFound: google-cloud-ndb"

At first it seems like the system didn’t install google-cloud-ndb correctly, but the real problem is with the version of setuptools that our instance was using, which as you can see from the traceback is setuptools-0.6c11. If you look at the Googles list of supported 3rd party libs, that version is deprecated.

The fix should be easy, update your app.yaml to use the latest setuptools (as of this writing its 36.6.0).

...
libraries:
- name: setuptools
  version: 36.6.0 # You can also use `latest`
...

Unfortunately for us, the problem still persisted and we kept seeing in our logs that we were still using the old setuptools. After more debugging, we realized that another dependency we had in app.yaml was requiring the old version. My senior pointed out that if you look at google/appengine/api/appinfo.py inside Google’s SDK, you’ll see the following:

REQUIRED_LIBRARIES = {
    ('django', '1.11'): [('pytz', '2017.2')],
    ('flask', '0.12'): [('click', '6.6'), ('itsdangerous', '0.24'),
                        ('jinja2', '2.6'), ('werkzeug', '0.11.10')],
    ('jinja2', '2.6'): [('markupsafe', '0.15'), ('setuptools', '0.6c11')],
    ('jinja2', 'latest'): [('markupsafe', 'latest'), ('setuptools', 'latest')],
    ('matplotlib', '1.2.0'): [('numpy', '1.6.1')],
    ('matplotlib', 'latest'): [('numpy', 'latest')],
    ('protobuf', '3.0.0'): [('six', 'latest')],
    ('protobuf', 'latest'): [('six', 'latest')],
    ('grpcio', '1.0.0'): [('protobuf', '3.0.0'), ('enum', '0.9.23'),
                          ('futures', '3.0.5'), ('six', 'latest'),
                          ('setuptools', '36.6.0')],
    ('grpcio', 'latest'): [('protobuf', 'latest'), ('enum', 'latest'),
                           ('futures', 'latest'), ('six', 'latest'),
                           ('setuptools', 'latest')]
}

We were using jinja2 2.6 which required the deprecated setuptools. We decided to remove jinja2 from app.yaml and vendor it with the rest of the external 3rd party libraries. After that, the issue was fixed!