A measurement of the associated production of a top-quark pair (t¯t) with a vector boson (W, Z) in proton-proton collisions at a center-of-mass energy of 13 TeV is presented, using 36.1??fb-1 of integrated luminosity collected by the ATLAS detector at the Large Hadron Collider. Events are selected in channels with two same- or opposite-sign leptons (electrons or muons), three leptons or four leptons, and each channel is further divided into multiple regions to maximize the sensitivity of the measurement. The t¯tZ and t¯tW production cross sections are simultaneously measured using a combined fit to all regions. The best-fit values of the production cross sections are st¯tZ=0.95±0.08stat±0.10syst?pb and st¯tW=0.87±0.13stat±0.14syst?pb in agreement with the Standard Model predictions. The measurement of the t¯tZ cross section is used to set constraints on effective field theory operators which modify the t¯tZ vertex.