Add support for YAML files, when PyYAML is installed. If PyYAML is not installed, any YAML file_data functions error out.

Only files ending with .yml or .yaml are considered to be YAML files. All other files are loaded as JSON files.

Dots is not allowed on function/methods names, so the names generated for ddt should not allow dots on the generated name.

This pull request ensure that there are no dots on ddt generated function/method names.

I noticed that @txels had expressed the desire to ditch backwards compatibility and give ddt another shot to get a more powerful platform on the other side. Hence, with the intent of sparking discussion, I write this.

I am officially dropping support for ddt. It's had its time, but now that I've transitioned to using py.test in my projects, it brings me no value and no longer use it.

If you however still see value in it and would like to step up as a maintainer, let me know and we'll manage a transfer of ownership.

Please comment on this issue if you're interested.

Here's a simple test case:

import unittest
from ddt import ddt, data

@ddt
class Test(unittest.TestCase):
    @data(1)
    def test_scalar(self, value):
        pass

    @data({"a": 1, "b": 2})
    def test_dict(self, value):
        pass

When run with python3.4which nosetests-v test.py you'll get (potentially) different test names with different runs:

$ nosetests -V
nosetests version 1.3.1
$ python3.4 `which nosetests` -v test.py
test_dict_1___b___2___a___1_ (test.Test) ... ok
test_scalar_1_1 (test.Test) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK
$ python3.4 `which nosetests` -v test.py
test_dict_1___a___1___b___2_ (test.Test) ... ok
test_scalar_1_1 (test.Test) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

Why this is a problem:

• It breaks nosetest's TestId functionality (e.g. makes running nosetests --with-ids --failed fail). Being able to re-run failed tests only (--failed) is a super-useful feature, so it not working is a big problem (time-waster).

The reason why this happens is that from Python 3.3 onwards hash randomization has been the default option (see https://docs.python.org/3.3/using/cmdline.html#cmdoption-R). This means that dict key hashes (at least for types with hash function depending on the system hash()) change from different runs, and thus the naming of tests using @data with dicts (or sets/frozensets for that matter) will vary from one run to another.

Although there is a simple workaround (use a fixed PYTHONHASHSEED when running tests) this is still not a completely good solution for a few reasons:

• This is going to bite a lot of people not familiar with hash randomization.
• This is going to bite those who move from older pythons to 3.3 or newer versions.

On the other hand I am not sure how to fix this. A few possibilities:

1. Update mk_test_name to recognize dicts (and sets and frozensets and recursive versions of these) and output keys in known (sorted) order.
2. Add a check that will produce a visible WARNING message when it detects running in post-3.3 python without PYTHONHASHSEED being set.
3. Add a check that will make mk_test_name omit the str(value) bit on >= 3.3 python if PYTHONHASHSEED hasn't been set. This would generate test names based only the running index, keeping them unique and immune to hash key randomization.

(And in any case add a note about this behavior to @data documentation.)

I think the first one is a can of worms since you'd need to handle recursive dicts and it really does not help with complex types that are not dicts but internally use dicts and produce str values from those.

The second one would help, but would essentially require changes to test environment when moving to >= 3.3 python on existing projects, and the warning might also be missed by a lot of people.

I think the third one would present the least surprise to people as it'd at least keep test cases accessible and running, yet makes it possible to get the full name (with test data str-encoded) by introducing PYTHONHASHSEED to the test environment by the user.

Comments, thoughts?

What: I would like to add a feature to ddt so that test names can be generated using just the index only.

How: To allow that to happen, I am making use of kwargs at the class-level decorator @ddt. When decorating a class with @ddt(formatTestName=FormatTestName.INDEX_ONLY), the boolean value will be passed to mk_test_name() and test names will just be index-only.

Why: An easier way to trigger a specific test to run.

Testing: New unit test added. Using tox to run against py27 and py35.

GLOB sdist-make: /path/masked/ddt/setup.py
py27 inst-nodeps: /path/masked/ddt/.tox/.tmp/package/1/ddt-1.3.1.zip
py27 installed: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support,alabaster==0.7.12,Babel==2.8.0,certifi==2020.4.5.1,chardet==3.0.4,configparser==4.0.2,coverage==5.1,ddt==1.3.1,docutils==0.16,entrypoints==0.3,enum34==1.1.10,flake8==3.7.9,funcsigs==1.0.2,functools32==3.2.3.post2,idna==2.9,imagesize==1.2.0,Jinja2==2.11.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,nose==1.3.7,packaging==20.3,pycodestyle==2.5.0,pyflakes==2.1.1,Pygments==2.5.2,pyparsing==2.4.7,pytz==2019.3,PyYAML==5.3.1,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,Sphinx==1.8.5,sphinxcontrib-programoutput==0.16,sphinxcontrib-websupport==1.1.2,typing==3.7.4.1,urllib3==1.25.8
py27 run-test-pre: PYTHONHASHSEED='2507378467'
py27 run-test: commands[0] | nosetests -s --with-coverage --cover-package=ddt --cover-html
.................................................................................
Name     Stmts   Miss  Cover
----------------------------
ddt.py     131      0   100%
----------------------------------------------------------------------
Ran 81 tests in 0.092s

OK
py27 run-test: commands[1] | flake8 ddt.py test
py27 run-test: commands[2] | sphinx-build -b html docs docs/_build
Running Sphinx v1.8.5
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: [] 0 added, 1 changed, 0 removed
reading sources... [100%] example

/path/masked/ddt/docs/example.rst:29: WARNING: Include file u'/path/masked/ddt/test/test_data_dict_dict.json' not found or reading it failed
/path/masked/ddt/docs/example.rst:34: WARNING: Include file u'/path/masked/ddt/test/test_data_dict_dict.yaml' not found or reading it failed
/path/masked/ddt/docs/example.rst:39: WARNING: Include file u'/path/masked/ddt/test/test_data_dict.json' not found or reading it failed
/path/masked/ddt/docs/example.rst:44: WARNING: Include file u'/path/masked/ddt/test/test_data_dict.yaml' not found or reading it failed
/path/masked/ddt/docs/example.rst:49: WARNING: Include file u'/path/masked/ddt/test/test_data_list.json' not found or reading it failed
/path/masked/ddt/docs/example.rst:54: WARNING: Include file u'/path/masked/ddt/test/test_data_list.yaml' not found or reading it failed
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 50%] example
writing output... [100%] index

generating indices... genindex py-modindex
writing additional pages... search
copying static files... WARNING: html_static_path entry u'/path/masked/ddt/docs/_static' does not exist
done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 7 warnings.

The HTML pages are in docs/_build.
py35 inst-nodeps: /path/masked/ddt/.tox/.tmp/package/1/ddt-1.3.1.zip
py35 installed: coverage==5.1,ddt==1.3.1,entrypoints==0.3,flake8==3.7.9,mccabe==0.6.1,nose==1.3.7,pycodestyle==2.5.0,pyflakes==2.1.1,PyYAML==5.3.1,six==1.14.0
py35 run-test-pre: PYTHONHASHSEED='2507378467'
py35 run-test: commands[0] | nosetests -s --with-coverage --cover-package=ddt --cover-html
.................................................................................
Name     Stmts   Miss  Cover
----------------------------
ddt.py     131      0   100%
----------------------------------------------------------------------
Ran 81 tests in 0.099s

OK
py35 run-test: commands[1] | flake8 ddt.py test
___________________________________________________________________________ summary ____________________________________________________________________________
  py27: commands succeeded
  py35: commands succeeded
  congratulations :)

Hi @txels

I've added a new decorator file_data that will load test data from a JSON file.

At the moment the JSON file can only contain a flat list of values but if you like this idea/approach I can add support to a schema like the following

[
 { 'test_name':  [1, 2, 5, 6],
   'test_name_2': [10, 33, 5]
 }
 ...
]

This would create 4 tests with the test_name appended to the original test function name (or even replaced but this might be confusing) and 3 more tests with test_name_2 appended to the name.

The purpose is to replace the construct below:

@data(*[a for a in foo if a > 0])

With this typing & memory-efficient alternative:

@data(a for a in foo if a > 0)

(*) It is a minor change that should not interfere with the possible re-write of the library in other branches.

A change in the latest release of ddt (#22 included in 1.0.1) has seemingly introduced breaking behavior for wrapped tests expecting a dictionary object when receiving test data, as exemplified below:

test_data.json

{
  "test_name": {
    "arg_one": "test1",
    "arg_two": "test2",
    "arg_three": "test3"
  }
}

Original test code:

@ddt.file_data('test_data.json')
def test_something(inputs):
    arg_one = inputs["arg_one"]
    arg_two = inputs["arg_two"]
    arg_three = inputs["arg_three"]
    # Perform tests here
    pass

New test code:

@ddt.file_data('test_data.json')
def test_something(arg_one, arg_two, arg_three):
    # Perform tests here
    pass

Since this behavior appears to be non-configurable and backward-incompatible, this seems more like a major version change rather than a simple patch. Can this change be reverted at least for the time being?

Right now, if you want to pass several arguments to your test, you typically provide a tuple for each test case, and have to unpack them inside the test method, such as:

@data((1,2,3), (4,5,9))
def test_things_with_3_numbers(self, arg):
    op1, op2, addition = arg
    ...

A nice improvement could be if ddt did this unpacking for you:

@packed_data((1,2,3), (4,5,9))
def test_things_with_3_numbers(self, op1, op2, addition):
    ...

Basically, this allows to do something like:

def make_data(...):
    d = ...
    setattr(d, "__name__", "friendly_name_computed_from_data")
    return d

@dds
class ...:
    @data(make_data(...), make_data(....), make_data(...))
    def test_much_data(self, data):
        ....

Unless __name__ is in the data, the current formatting behavior is used.

The python-six dependency is only used to detect python version in one test. This patch uses the pytest.mark.skipif to split that test in two and skip dependending on python version.

@named_data currently explicitly checks for lists or dicts, but in the case of lists it could just check against sequences (including tuples) and work in exactly the same way.

tldr; a combination of the @idata and @file_data decorators

Use Case

Add a decorator @ifile_data to pass multiple files as data items into a test function.

1. Files would be specified by a glob pattern relative to the test files directory.

2. The name of the test case would be the file.name of each file.

3. Test generated test names should include the file name.

MyTest_0_some_1234_json MyTest_1_some_xyz_json

an example of how this operator would be used.

@ddt
class MyTestCase(unittest.TestCase):

  @ifile_data('./data/some_*.json')
  def MyTest(self, some_data):
    self.assertIsNotNone(some_data)

workaround

I am using this as a workaround for now:

from unittest import TestCase, mock
import json
from ddt import ddt, idata
from pathlib import Path


class NamedList(list):
    pass


def get_file_data(glob_pattern):

    result = []

    for file in Path(__file__).parent.glob(glob_pattern):
        with open(file) as reader:
            dataitem = NamedList(json.load(reader))
            setattr(dataitem, '__name__', file.name)
            result.append(dataitem)

    return result


@ddt
class FileDataTests(TestCase):
    @idata(get_file_data('./data/some_*.json'))
    def test_file_data(self, some_data):
        self.assertIsNotNone(some_data)

With unittest it is possible to run specific tests from the cli. When using ddt this doesn't seem to be possible.

Example:

import unittest
from ddt import data, ddt


class Test(unittest.TestCase):
    def test(self):
        self.assertEqual(1, 1)

@ddt
class TestDDT(unittest.TestCase):
    @data('arg1', 'arg2')
    def test(self, arg='arg1'):
        self.assertEqual(1, 1)

if __name__ == '__main__':
    unittest.main(exit=False)

Running specific unittest:

$  python test.py Test.test
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Running specific unittest with ddt:

$ python test.py TestDDT.test
E
======================================================================
ERROR: test (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: type object 'TestDDT' has no attribute 'test'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)