diff --git a/ChangeLog.md b/ChangeLog.md index 0912ce7dd4..1e082c575b 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -66,6 +66,7 @@ Major changes: * Remove support for building GHCJS itself. Future releases of Stack may remove GHCJS support entirely. +* Support for lock files for pinning exact project dependency versions Behavior changes: * `stack.yaml` now supports `snapshot`: a synonym for `resolver`. See [#4256](https://github.com/commercialhaskell/stack/issues/4256) diff --git a/doc/lock_files.md b/doc/lock_files.md index fc6260158b..fee779ccd5 100644 --- a/doc/lock_files.md +++ b/doc/lock_files.md @@ -18,11 +18,12 @@ set of input files. There are a few problems with making this work: To address this, we follow the (fairly standard) approach of having a _lock file_. The goal of the lock file is to cache completed -information about all packages and snapshot files so that: +locations of project, snapshot packages and snapshots themselves so that: * These files can be stored in source control * Users on other machines can reuse these lock files and get identical - build plans + build plans given that the used local packages and local snapshots are + the same on those machines * Rerunning `stack build` in the future is deterministic in the build plan, not depending on mutable state in the world like Hackage revisions @@ -31,8 +32,6 @@ information about all packages and snapshot files so that: to perform the build. However, by deterministic, we mean it either performs the same build or fails, never accidentally doing something different. -* Stack can quickly determine the build plan in the common case of no - changes to `stack.yaml` or snapshot files This document explains the contents of a lock file, how they are used, and how they are created and updated. @@ -42,11 +41,7 @@ and how they are created and updated. Relevant to this discussion, the `stack.yaml` file specifies: * Resolver (the parent snapshot) -* Compiler override * `extra-deps` -* Flags -* GHC options -* Hidden packages The resolver can either specify a compiler version or another snapshot file. This snapshot file can contain the same information referenced @@ -55,12 +50,7 @@ above for a `stack.yaml`, with the following differences: * The `extra-deps` are called `packages` * Drop packages can be included -Some of this information is, by its nature, complete. For example, the -"flags" field cannot be influenced by anything outside of the file -itself. - -On the other hand, some information in these files can be -incomplete. Consider: +Some information in these files can be incomplete. Consider: ```yaml resolver: lts-13.9 @@ -128,24 +118,16 @@ parsing of the additional files in the common case of no changes. The lock file contains the following information: -* The full snapshot definition information, including completed - package locations for both `extra-deps` and packages in +* Completed package locations for both `extra-deps` and packages in snapshot files * **NOTE** This only applies to _immutable_ packages. Mutable packages are not included in the lock file. * Completed information for the snapshot locations -* A hash of the `stack.yaml` file -* The snapshot hash, to bypass the need to recalculate this on each - run of Stack It looks like the following: ```yaml # Lock file, some message about the file being auto-generated -stack-yaml: - sha256: XXXX - size: XXXX # in bytes - snapshots: # Starts with the snapshot specified in stack.yaml, # then continues with the snapshot specified in each @@ -163,33 +145,22 @@ snapshots: url: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/13/9.yaml sha256: 83de9017d911cf7795f19353dba4d04bd24cd40622b7567ff61fc3f7223aa3ea -compiler: ghc-X.Y.Z - packages: - acme-missiles: - location: - # QUESTION: any reason we need to specify which snapshot file it came from? I don't think so... - original: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz - completed: - size: 1442 - url: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz - cabal-file: - size: 613 - sha256: 2ba66a092a32593880a87fb00f3213762d7bca65a687d45965778deb8694c5d1 - name: acme-missiles - version: '0.3' - sha256: e563d8b524017a06b32768c4db8eff1f822f3fb22a90320b7e414402647b735b - pantry-tree: - size: 226 - sha256: 614bc0cca76937507ea0a5ccc17a504c997ce458d7f2f9e43b15a10c8eaeb033 - flags: ... - hidden: true/false - ghc-options: [...] +- original: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz + completed: + size: 1442 + url: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz + cabal-file: + size: 613 + sha256: 2ba66a092a32593880a87fb00f3213762d7bca65a687d45965778deb8694c5d1 + name: acme-missiles + version: '0.3' + sha256: e563d8b524017a06b32768c4db8eff1f822f3fb22a90320b7e414402647b735b + pantry-tree: + size: 226 + sha256: 614bc0cca76937507ea0a5ccc17a504c997ce458d7f2f9e43b15a10c8eaeb033 ``` -**NOTE** The `original` fields may seem superfluous at first. See the -update procedure below for an explanation. - ## Creation Whenever a `stack.yaml` file is loaded, Stack checks for a lock file @@ -206,36 +177,9 @@ If the lock file does not exist, it will be created by: * Completing all missing information * Writing out the new `stack.yaml.lock` file -## Dirtiness checking - -If the `stack.yaml.lock` file exists, its last modification time is -compared against the last modification time of the `stack.yaml` file -and any local snapshot files. If any of those files is more recent -than the `stack.yaml` file, and the file hashes in the lock file -do not match the files on the filesystem, then the update procedure is -triggered. Otherwise, the `stack.yaml.lock` file can be used as the -definition of the snapshot. - ## Update procedure -The simplest possible implementation is: ignore the lock file entirely -and create a new one followign the creation steps above. There's a -significant downside to this, however: it may cause a larger delta in -the lock file than intended, by causing more packages to be -updates. For example, many packages from Hackage may have their -Hackage revision information updated unnecessarily. - -The more complicated update procedure is described below. **QUESTION** -Do we want to go the easy way at first and later implement the more -complicated update procedure? - -1. Create a map from original package location to completed package - location in the lock file -2. Load up each snapshot file -3. For each incomplete package location: - * Lookup the value in the map created in (1) - * If present: use that completed information - * Otherwise: complete the information using the same completion - procedure from Pantry as in "creation" - -This should minimize the number of changes to packages incurred. +When loading a Stack project all completed package or snapshot locations +(even when they were completed using information from a lock file) get +collected to form a new lock file in memory and compare against the one +on disk, writing if there are any differences. diff --git a/package.yaml b/package.yaml index e2fd0fa72b..47d2db1670 100644 --- a/package.yaml +++ b/package.yaml @@ -183,6 +183,7 @@ library: - Stack.IDE - Stack.Init - Stack.Ls + - Stack.Lock - Stack.New - Stack.Nix - Stack.Options.BenchParser @@ -302,6 +303,7 @@ tests: dependencies: - QuickCheck - hspec + - raw-strings-qq - stack - smallcheck flags: diff --git a/src/Stack/Config.hs b/src/Stack/Config.hs index fd726a4e57..539f5a5e45 100644 --- a/src/Stack/Config.hs +++ b/src/Stack/Config.hs @@ -62,6 +62,7 @@ import Stack.Config.Docker import Stack.Config.Nix import Stack.Constants import Stack.Build.Haddock (shouldHaddockDeps) +import Stack.Lock (lockCachedWanted) import Stack.Storage (initStorage) import Stack.SourceMap import Stack.Types.Build @@ -76,6 +77,7 @@ import System.Console.ANSI (hSupportsANSIWithoutEmulation) import System.Environment import System.PosixCompat.Files (fileOwner, getFileStatus) import System.PosixCompat.User (getEffectiveUserID) +import RIO.List (unzip) import RIO.PrettyPrint (stylesUpdateL, useColorL) import RIO.Process @@ -503,12 +505,51 @@ loadBuildConfig = do { projectCompiler = mcompiler <|> projectCompiler project' , projectResolver = fromMaybe (projectResolver project') mresolver } + extraPackageDBs <- mapM resolveDir' (projectExtraPackageDBs project) - resolver <- completeSnapshotLocation $ projectResolver project - (snapshot, _completed) <- loadAndCompleteSnapshot resolver + wanted <- lockCachedWanted stackYamlFP (projectResolver project) $ + fillProjectWanted stackYamlFP config project - extraPackageDBs <- mapM resolveDir' (projectExtraPackageDBs project) + return BuildConfig + { bcConfig = config + , bcSMWanted = wanted + , bcExtraPackageDBs = extraPackageDBs + , bcStackYaml = stackYamlFP + , bcCurator = projectCurator project + } + where + getEmptyProject :: Maybe RawSnapshotLocation -> [PackageIdentifierRevision] -> RIO Config Project + getEmptyProject mresolver extraDeps = do + r <- case mresolver of + Just resolver -> do + logInfo ("Using resolver: " <> display resolver <> " specified on command line") + return resolver + Nothing -> do + r'' <- getLatestResolver + logInfo ("Using latest snapshot resolver: " <> display r'') + return r'' + return Project + { projectUserMsg = Nothing + , projectPackages = [] + , projectDependencies = map (RPLImmutable . flip RPLIHackage Nothing) extraDeps + , projectFlags = mempty + , projectResolver = r + , projectCompiler = Nothing + , projectExtraPackageDBs = [] + , projectCurator = Nothing + , projectDropPackages = mempty + } +fillProjectWanted :: + (HasProcessContext env, HasLogFunc env, HasPantryConfig env) + => Path Abs t + -> Config + -> Project + -> Map RawPackageLocationImmutable PackageLocationImmutable + -> WantedCompiler + -> Map PackageName (Bool -> RIO env DepPackage) + -> RIO env (SMWanted, [CompletedPLI]) +fillProjectWanted stackYamlFP config project locCache snapCompiler snapPackages = do let bopts = configBuild config packages0 <- for (projectPackages project) $ \fp@(RelFilePath t) -> do @@ -517,25 +558,27 @@ loadBuildConfig = do pp <- mkProjectPackage YesPrintWarnings resolved (boptsHaddock bopts) pure (cpName $ ppCommon pp, pp) - let completeLocation (RPLMutable m) = pure $ PLMutable m - completeLocation (RPLImmutable im) = PLImmutable <$> completePackageLocation im - - deps0 <- forM (projectDependencies project) $ \rpl -> do - pl <- completeLocation rpl + (deps0, mcompleted) <- fmap unzip . forM (projectDependencies project) $ \rpl -> do + (pl, mCompleted) <- case rpl of + RPLImmutable rpli -> do + compl <- maybe (completePackageLocation rpli) pure (Map.lookup rpli locCache) + pure (PLImmutable compl, Just (rpli, compl)) + RPLMutable p -> + pure (PLMutable p, Nothing) dp <- additionalDepPackage (shouldHaddockDeps bopts) pl - pure (cpName $ dpCommon dp, dp) + pure ((cpName $ dpCommon dp, dp), mCompleted) checkDuplicateNames $ map (second (PLMutable . ppResolvedDir)) packages0 ++ map (second dpLocation) deps0 let packages1 = Map.fromList packages0 - snPackages = snapshotPackages snapshot + snPackages = snapPackages `Map.difference` packages1 `Map.difference` Map.fromList deps0 `Map.withoutKeys` projectDropPackages project - snDeps <- Map.traverseWithKey (snapToDepPackage (shouldHaddockDeps bopts)) snPackages + snDeps <- for snPackages $ \getDep -> getDep (shouldHaddockDeps bopts) let deps1 = Map.fromList deps0 `Map.union` snDeps @@ -561,41 +604,14 @@ loadBuildConfig = do throwM $ InvalidGhcOptionsSpecification (Map.keys unusedPkgGhcOptions) let wanted = SMWanted - { smwCompiler = fromMaybe (snapshotCompiler snapshot) (projectCompiler project) + { smwCompiler = fromMaybe snapCompiler (projectCompiler project) , smwProject = packages , smwDeps = deps , smwSnapshotLocation = projectResolver project } - return BuildConfig - { bcConfig = config - , bcSMWanted = wanted - , bcExtraPackageDBs = extraPackageDBs - , bcStackYaml = stackYamlFP - , bcCurator = projectCurator project - } - where - getEmptyProject :: Maybe RawSnapshotLocation -> [PackageIdentifierRevision] -> RIO Config Project - getEmptyProject mresolver extraDeps = do - r <- case mresolver of - Just resolver -> do - logInfo ("Using resolver: " <> display resolver <> " specified on command line") - return resolver - Nothing -> do - r'' <- getLatestResolver - logInfo ("Using latest snapshot resolver: " <> display r'') - return r'' - return Project - { projectUserMsg = Nothing - , projectPackages = [] - , projectDependencies = map (RPLImmutable . flip RPLIHackage Nothing) extraDeps - , projectFlags = mempty - , projectResolver = r - , projectCompiler = Nothing - , projectExtraPackageDBs = [] - , projectCurator = Nothing - , projectDropPackages = mempty - } + pure (wanted, catMaybes mcompleted) + -- | Check if there are any duplicate package names and, if so, throw an -- exception. diff --git a/src/Stack/Freeze.hs b/src/Stack/Freeze.hs index c751b3145d..77fa033a0c 100644 --- a/src/Stack/Freeze.hs +++ b/src/Stack/Freeze.hs @@ -69,7 +69,7 @@ doFreeze p FreezeSnapshot = do case result of Left _wc -> logInfo "No freezing is required for compiler resolver" - Right (snap, _) -> do + Right snap -> do snap' <- completeSnapshotLayer snap let rawCompleted = toRawSnapshotLayer snap' if rawCompleted == snap diff --git a/src/Stack/Lock.hs b/src/Stack/Lock.hs new file mode 100644 index 0000000000..bb9709a82d --- /dev/null +++ b/src/Stack/Lock.hs @@ -0,0 +1,125 @@ +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE FlexibleInstances #-} +{-# LANGUAGE NoImplicitPrelude #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE RecordWildCards #-} + +module Stack.Lock + ( lockCachedWanted + , LockedLocation(..) + , Locked(..) + ) where + +import Data.Aeson.Extended +import qualified Data.List.NonEmpty as NE +import Data.Map (Map) +import qualified Data.Map as Map +import qualified Data.Yaml as Yaml +import Pantry +import Path (addFileExtension, parent) +import Path.IO (doesFileExist) +import RIO.Process +import Stack.Prelude +import Stack.SourceMap +import Stack.Types.Config +import Stack.Types.SourceMap + +data LockedLocation a b = LockedLocation + { llOriginal :: a + , llCompleted :: b + } deriving (Eq, Show) + +instance (ToJSON a, ToJSON b) => ToJSON (LockedLocation a b) where + toJSON ll = + object [ "original" .= llOriginal ll, "completed" .= llCompleted ll ] + +instance ( FromJSON (WithJSONWarnings (Unresolved a)) + , FromJSON (WithJSONWarnings (Unresolved b)) + ) => + FromJSON (WithJSONWarnings (Unresolved (LockedLocation a b))) where + parseJSON = + withObjectWarnings "LockedLocation" $ \o -> do + original <- jsonSubWarnings $ o ..: "original" + completed <- jsonSubWarnings $ o ..: "completed" + pure $ LockedLocation <$> original <*> completed + +-- Special wrapper extracting only 1 RawPackageLocationImmutable +-- serialization should not produce locations with multiple subdirs +-- so we should be OK using just a head element +newtype SingleRPLI = SingleRPLI { unSingleRPLI :: RawPackageLocationImmutable} + +instance FromJSON (WithJSONWarnings (Unresolved SingleRPLI)) where + parseJSON v = + do + WithJSONWarnings unresolvedRPLIs ws <- parseJSON v + let withWarnings x = WithJSONWarnings x ws + pure $ withWarnings $ SingleRPLI . NE.head <$> unresolvedRPLIs + +data Locked = Locked + { lckSnapshotLocaitons :: [LockedLocation RawSnapshotLocation SnapshotLocation] + , lckPkgImmutableLocations :: [LockedLocation RawPackageLocationImmutable PackageLocationImmutable] + } deriving (Eq, Show) + +instance ToJSON Locked where + toJSON Locked {..} = + object + [ "snapshots" .= lckSnapshotLocaitons + , "packages" .= lckPkgImmutableLocations + ] + +instance FromJSON (WithJSONWarnings (Unresolved Locked)) where + parseJSON = withObjectWarnings "Locked" $ \o -> do + snapshots <- jsonSubWarningsT $ o ..: "snapshots" + packages <- jsonSubWarningsT $ o ..: "packages" + let unwrap ll = ll { llOriginal = unSingleRPLI (llOriginal ll) } + pure $ Locked <$> sequenceA snapshots <*> (map unwrap <$> sequenceA packages) + +loadYamlThrow + :: HasLogFunc env + => (Value -> Yaml.Parser (WithJSONWarnings a)) -> Path Abs File -> RIO env a +loadYamlThrow parser path = do + val <- liftIO $ Yaml.decodeFileThrow (toFilePath path) + case Yaml.parseEither parser val of + Left err -> throwIO $ Yaml.AesonException err + Right (WithJSONWarnings res warnings) -> do + logJSONWarnings (toFilePath path) warnings + return res + +lockCachedWanted :: + (HasPantryConfig env, HasProcessContext env, HasLogFunc env) + => Path Abs File + -> RawSnapshotLocation + -> (Map RawPackageLocationImmutable PackageLocationImmutable + -> WantedCompiler + -> Map PackageName (Bool -> RIO env DepPackage) + -> RIO env ( SMWanted, [CompletedPLI])) + -> RIO env SMWanted +lockCachedWanted stackFile resolver fillWanted = do + lockFile <- liftIO $ addFileExtension "lock" stackFile + lockExists <- doesFileExist lockFile + locked <- + if not lockExists + then do + logDebug "Lock file doesn't exist" + pure $ Locked [] [] + else do + logDebug "Using package location completions from a lock file" + unresolvedLocked <- loadYamlThrow parseJSON lockFile + resolvePaths (Just $ parent stackFile) unresolvedLocked + let toMap :: Ord a => [LockedLocation a b] -> Map a b + toMap = Map.fromList . map (\ll -> (llOriginal ll, llCompleted ll)) + slocCache = toMap $ lckSnapshotLocaitons locked + pkgLocCache = toMap $ lckPkgImmutableLocations locked + (snap, slocCompleted, pliCompleted) <- + loadAndCompleteSnapshotRaw resolver slocCache pkgLocCache + let compiler = snapshotCompiler snap + snPkgs = Map.mapWithKey (\n p h -> snapToDepPackage h n p) (snapshotPackages snap) + (wanted, prjCompleted) <- fillWanted Map.empty compiler snPkgs + let lockLocations = map (uncurry LockedLocation) + newLocked = Locked { lckSnapshotLocaitons = lockLocations slocCompleted + , lckPkgImmutableLocations = + lockLocations $ pliCompleted <> prjCompleted + } + when (newLocked /= locked) $ + liftIO $ Yaml.encodeFile (toFilePath lockFile) newLocked + pure wanted diff --git a/src/Stack/SourceMap.hs b/src/Stack/SourceMap.hs index 98a4346ff3..6e8070e219 100644 --- a/src/Stack/SourceMap.hs +++ b/src/Stack/SourceMap.hs @@ -262,7 +262,7 @@ loadProjectSnapshotCandidate :: -> Bool -> RIO env (SnapshotCandidate env) loadProjectSnapshotCandidate loc printWarnings buildHaddocks = do - snapshot <- fmap fst . loadAndCompleteSnapshot =<< completeSnapshotLocation loc + (snapshot, _, _) <- loadAndCompleteSnapshotRaw loc Map.empty Map.empty deps <- Map.traverseWithKey (snapToDepPackage False) (snapshotPackages snapshot) let wc = snapshotCompiler snapshot globals <- Map.map GlobalPackageVersion <$> globalsFromHints wc diff --git a/src/test/Stack/LockSpec.hs b/src/test/Stack/LockSpec.hs new file mode 100644 index 0000000000..3d15829987 --- /dev/null +++ b/src/test/Stack/LockSpec.hs @@ -0,0 +1,172 @@ +{-# LANGUAGE QuasiQuotes #-} +{-# LANGUAGE ScopedTypeVariables #-} +{-# LANGUAGE OverloadedStrings #-} + +module Stack.LockSpec where + +import Data.Aeson.Extended (WithJSONWarnings(..)) +import Data.ByteString (ByteString) +import qualified Data.Yaml as Yaml +import Distribution.Types.PackageName (mkPackageName) +import Distribution.Types.Version (mkVersion) +import Pantry +import qualified Pantry.SHA256 as SHA256 +import RIO +import Stack.Lock +import Test.Hspec +import Text.RawString.QQ + +toBlobKey :: ByteString -> Word -> BlobKey +toBlobKey string size = BlobKey (decodeSHA string) (FileSize size) + +decodeSHA :: ByteString -> SHA256 +decodeSHA string = + case SHA256.fromHexBytes string of + Right csha -> csha + Left err -> error $ "Failed decoding. Error: " <> show err + +decodeLocked :: ByteString -> IO Locked +decodeLocked bs = do + val <- Yaml.decodeThrow bs + case Yaml.parseEither Yaml.parseJSON val of + Left err -> throwIO $ Yaml.AesonException err + Right (WithJSONWarnings res warnings) -> do + unless (null warnings) $ + throwIO $ Yaml.AesonException $ "Unexpected warnings: " ++ show warnings + -- we just assume no file references + resolvePaths Nothing res + +spec :: Spec +spec = do + it "parses lock file (empty with GHC resolver)" $ do + let lockFile :: ByteString + lockFile = + [r|#some +snapshots: +- completed: + compiler: ghc-8.2.2 + original: + compiler: ghc-8.2.2 +packages: [] +|] + pkgImm <- lckPkgImmutableLocations <$> decodeLocked lockFile + pkgImm `shouldBe` [] + it "parses lock file (empty with LTS resolver)" $ do + let lockFile :: ByteString + lockFile = + [r|#some +snapshots: +- completed: + size: 527801 + url: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/11/22.yaml + sha256: 7c8b1853da784bd7beb8728168bf4e879d8a2f6daf408ca0fa7933451864a96a + original: lts-11.22 +- completed: + compiler: ghc-8.2.2 + original: + compiler: ghc-8.2.2 +packages: [] +|] + pkgImm <- lckPkgImmutableLocations <$> decodeLocked lockFile + pkgImm `shouldBe` [] + it "parses lock file (LTS, wai + warp)" $ do + let lockFile :: ByteString + lockFile = + [r|#some +snapshots: +- completed: + size: 527801 + url: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/11/22.yaml + sha256: 7c8b1853da784bd7beb8728168bf4e879d8a2f6daf408ca0fa7933451864a96a + original: lts-11.22 +- completed: + compiler: ghc-8.2.2 + original: + compiler: ghc-8.2.2 +packages: +- original: + subdir: wai + git: https://github.com/yesodweb/wai.git + commit: d11d63f1a6a92db8c637a8d33e7953ce6194a3e0 + completed: + subdir: wai + cabal-file: + size: 1765 + sha256: eea52c4967d8609c2f79213d6dffe6d6601034f1471776208404781de7051410 + name: wai + version: 3.2.1.2 + git: https://github.com/yesodweb/wai.git + pantry-tree: + size: 714 + sha256: ecfd0b4b75f435a3f362394807b35e5ef0647b1a25005d44a3632c49db4833d2 + commit: d11d63f1a6a92db8c637a8d33e7953ce6194a3e0 +- original: + subdir: warp + git: https://github.com/yesodweb/wai.git + commit: d11d63f1a6a92db8c637a8d33e7953ce6194a3e0 + completed: + subdir: warp + cabal-file: + size: 10725 + sha256: cfec5336260bb4b1ecbd833f7d6948fd1ee373770471fe79796cf9c389c71758 + name: warp + version: 3.2.25 + git: https://github.com/yesodweb/wai.git + pantry-tree: + size: 5103 + sha256: f808e075811b002563d24c393ce115be826bb66a317d38da22c513ee42b7443a + commit: d11d63f1a6a92db8c637a8d33e7953ce6194a3e0 +|] + pkgImm <- lckPkgImmutableLocations <$> decodeLocked lockFile + let waiSubdirRepo subdir = + Repo { repoType = RepoGit + , repoUrl = "https://github.com/yesodweb/wai.git" + , repoCommit = + "d11d63f1a6a92db8c637a8d33e7953ce6194a3e0" + , repoSubdir = subdir + } + emptyRPM = RawPackageMetadata { rpmName = Nothing + , rpmVersion = Nothing + , rpmTreeKey = Nothing + , rpmCabal = Nothing + } + pkgImm `shouldBe` + [ LockedLocation + (RPLIRepo (waiSubdirRepo "wai") emptyRPM) + (PLIRepo (waiSubdirRepo "wai") + (PackageMetadata { pmIdent = + PackageIdentifier + { pkgName = mkPackageName "wai" + , pkgVersion = mkVersion [3, 2, 1, 2] + } + , pmTreeKey = + TreeKey + (BlobKey + (decodeSHA + "ecfd0b4b75f435a3f362394807b35e5ef0647b1a25005d44a3632c49db4833d2") + (FileSize 714)) + , pmCabal = + toBlobKey + "eea52c4967d8609c2f79213d6dffe6d6601034f1471776208404781de7051410" + 1765 + })) + , LockedLocation + (RPLIRepo (waiSubdirRepo "warp") emptyRPM) + (PLIRepo (waiSubdirRepo "warp") + (PackageMetadata { pmIdent = + PackageIdentifier + { pkgName = mkPackageName "warp" + , pkgVersion = mkVersion [3, 2, 25] + } + , pmTreeKey = + TreeKey + (BlobKey + (decodeSHA + "f808e075811b002563d24c393ce115be826bb66a317d38da22c513ee42b7443a") + (FileSize 5103)) + , pmCabal = + toBlobKey + "cfec5336260bb4b1ecbd833f7d6948fd1ee373770471fe79796cf9c389c71758" + 10725 + })) + ] diff --git a/subs/curator/app/Main.hs b/subs/curator/app/Main.hs index d5d9357139..9db9561e1f 100644 --- a/subs/curator/app/Main.hs +++ b/subs/curator/app/Main.hs @@ -156,8 +156,10 @@ snapshot = do loadSnapshotYaml :: RIO PantryApp Pantry.Snapshot loadSnapshotYaml = do abs' <- resolveFile' snapshotFilename - fmap fst $ loadAndCompleteSnapshot $ SLFilePath $ - ResolvedPath (RelFilePath (fromString snapshotFilename)) abs' + let sloc = SLFilePath $ + ResolvedPath (RelFilePath (fromString snapshotFilename)) abs' + (snap, _, _) <- loadAndCompleteSnapshot sloc Map.empty Map.empty + pure snap checkSnapshot :: RIO PantryApp () checkSnapshot = do @@ -219,4 +221,4 @@ loadPantrySnapshotLayerFile fp = do eres <- loadSnapshotLayer $ SLFilePath (ResolvedPath (RelFilePath (fromString fp)) abs') case eres of Left x -> error $ "should not happen: " ++ show (fp, x) - Right (x, _) -> pure x + Right x -> pure x diff --git a/subs/pantry/package.yaml b/subs/pantry/package.yaml index 57ad825507..dd706ebeb7 100644 --- a/subs/pantry/package.yaml +++ b/subs/pantry/package.yaml @@ -123,3 +123,4 @@ tests: - exceptions - hedgehog - QuickCheck + - raw-strings-qq diff --git a/subs/pantry/src/Pantry.hs b/subs/pantry/src/Pantry.hs index cef5a9ecc0..94087594c5 100644 --- a/subs/pantry/src/Pantry.hs +++ b/subs/pantry/src/Pantry.hs @@ -92,6 +92,8 @@ module Pantry , loadSnapshotLayer , loadSnapshot , loadAndCompleteSnapshot + , loadAndCompleteSnapshotRaw + , CompletedPLI , addPackagesToSnapshot , AddPackagesConfig (..) @@ -104,6 +106,7 @@ module Pantry , parseWantedCompiler , parseRawSnapshotLocation , parsePackageIdentifierRevision + , parseHackageText -- ** Cabal values , parsePackageIdentifier @@ -168,6 +171,7 @@ module Pantry import RIO import Conduit +import Control.Arrow (right) import Control.Monad.State.Strict (State, execState, get, modify') import qualified RIO.Map as Map import qualified RIO.Set as Set @@ -396,7 +400,7 @@ loadCabalFileImmutable loadCabalFileImmutable loc = withCache $ do logDebug $ "Parsing cabal file for " <> display loc bs <- loadCabalFileBytes loc - let foundCabalKey = BlobKey (SHA256.hashBytes bs) (FileSize (fromIntegral (B.length bs))) + let foundCabalKey = bsToBlobKey bs (_warnings, gpd) <- rawParseGPD (Left $ toRawPLI loc) bs let pm = case loc of @@ -442,7 +446,7 @@ loadCabalFileRawImmutable loadCabalFileRawImmutable loc = withCache $ do logDebug $ "Parsing cabal file for " <> display loc bs <- loadRawCabalFileBytes loc - let foundCabalKey = BlobKey (SHA256.hashBytes bs) (FileSize (fromIntegral (B.length bs))) + let foundCabalKey = bsToBlobKey bs (_warnings, gpd) <- rawParseGPD (Left loc) bs let rpm = case loc of @@ -796,8 +800,7 @@ completeSnapshotLocation (RSLFilePath f) = pure $ SLFilePath f completeSnapshotLocation (RSLUrl url (Just blobKey)) = pure $ SLUrl url blobKey completeSnapshotLocation (RSLUrl url Nothing) = do bs <- loadFromURL url Nothing - let blobKey = BlobKey (SHA256.hashBytes bs) (FileSize $ fromIntegral $ B.length bs) - pure $ SLUrl url blobKey + pure $ SLUrl url (bsToBlobKey bs) -- | Fill in optional fields in a 'SnapshotLayer' for more reproducible builds. -- @@ -906,7 +909,7 @@ loadSnapshotRaw loc = do , rsPackages = mempty , rsDrop = mempty } - Right (rsl, _sha) -> do + Right (rsl, _) -> do snap0 <- loadSnapshotRaw $ rslParent rsl (packages, unused) <- addPackagesToSnapshot @@ -942,7 +945,7 @@ loadSnapshot loc = do , rsPackages = mempty , rsDrop = mempty } - Right (rsl, _sha) -> do + Right rsl -> do snap0 <- loadSnapshotRaw $ rslParent rsl (packages, unused) <- addPackagesToSnapshot @@ -963,6 +966,7 @@ loadSnapshot loc = do } type CompletedPLI = (RawPackageLocationImmutable, PackageLocationImmutable) +type CompletedSL = (RawSnapshotLocation, SnapshotLocation) -- | Parse a 'Snapshot' (all layers) from a 'SnapshotLocation' noting -- any incomplete package locations @@ -971,9 +975,11 @@ type CompletedPLI = (RawPackageLocationImmutable, PackageLocationImmutable) loadAndCompleteSnapshot :: (HasPantryConfig env, HasLogFunc env, HasProcessContext env) => SnapshotLocation - -> RIO env (Snapshot, [CompletedPLI]) -loadAndCompleteSnapshot loc = - loadAndCompleteSnapshotRaw (toRawSL loc) + -> Map RawSnapshotLocation SnapshotLocation -- ^ Cached snapshot locations from lock file + -> Map RawPackageLocationImmutable PackageLocationImmutable -- ^ Cached locations from lock file + -> RIO env (Snapshot, [CompletedSL], [CompletedPLI]) +loadAndCompleteSnapshot loc cachedSL cachedPL = + loadAndCompleteSnapshotRaw (toRawSL loc) cachedSL cachedPL -- | Parse a 'Snapshot' (all layers) from a 'RawSnapshotLocation' completing -- any incomplete package locations @@ -982,9 +988,13 @@ loadAndCompleteSnapshot loc = loadAndCompleteSnapshotRaw :: (HasPantryConfig env, HasLogFunc env, HasProcessContext env) => RawSnapshotLocation - -> RIO env (Snapshot, [CompletedPLI]) -loadAndCompleteSnapshotRaw loc = do - eres <- loadRawSnapshotLayer loc + -> Map RawSnapshotLocation SnapshotLocation -- ^ Cached snapshot locations from lock file + -> Map RawPackageLocationImmutable PackageLocationImmutable -- ^ Cached locations from lock file + -> RIO env (Snapshot, [CompletedSL], [CompletedPLI]) +loadAndCompleteSnapshotRaw rawLoc cacheSL cachePL = do + eres <- case Map.lookup rawLoc cacheSL of + Just loc -> right (\rsl -> (rsl, (rawLoc, loc))) <$> loadSnapshotLayer loc + Nothing -> loadRawSnapshotLayer rawLoc case eres of Left wc -> let snapshot = Snapshot @@ -992,12 +1002,13 @@ loadAndCompleteSnapshotRaw loc = do , snapshotPackages = mempty , snapshotDrop = mempty } - in pure (snapshot, []) - Right (rsl, _sha) -> do - (snap0, completed0) <- loadAndCompleteSnapshotRaw $ rslParent rsl + in pure (snapshot, [(RSLCompiler wc, SLCompiler wc)], []) + Right (rsl, sloc) -> do + (snap0, slocs, completed0) <- loadAndCompleteSnapshotRaw (rslParent rsl) cacheSL cachePL (packages, completed, unused) <- addAndCompletePackagesToSnapshot - (display loc) + rawLoc + cachePL (rslLocations rsl) AddPackagesConfig { apcDrop = rslDropPackages rsl @@ -1006,13 +1017,13 @@ loadAndCompleteSnapshotRaw loc = do , apcGhcOptions = rslGhcOptions rsl } (snapshotPackages snap0) - warnUnusedAddPackagesConfig (display loc) unused + warnUnusedAddPackagesConfig (display rawLoc) unused let snapshot = Snapshot { snapshotCompiler = fromMaybe (snapshotCompiler snap0) (rslCompiler rsl) , snapshotPackages = packages , snapshotDrop = apcDrop unused } - return (snapshot, completed ++ completed0) + return (snapshot, sloc : slocs,completed0 ++ completed) data SingleOrNot a = Single !a @@ -1126,6 +1137,16 @@ addPackagesToSnapshot source newPackages (AddPackagesConfig drops flags hiddens pure (allPackages, unused) +cachedSnapshotCompletePackageLocation :: (HasPantryConfig env, HasLogFunc env, HasProcessContext env) + => Map RawPackageLocationImmutable PackageLocationImmutable + -> RawPackageLocationImmutable + -> RIO env PackageLocationImmutable +cachedSnapshotCompletePackageLocation cachePackages rpli = do + let xs = Map.lookup rpli cachePackages + case xs of + Nothing -> completePackageLocation rpli + Just x -> pure x + -- | Add more packages to a snapshot completing their locations if needed -- -- Note that any settings on a parent flag which is being replaced will be @@ -1133,31 +1154,39 @@ addPackagesToSnapshot source newPackages (AddPackagesConfig drops flags hiddens -- set, and @foo@ also appears in new packages, then @bar@ will no longer be -- set. -- --- Returns any of the 'AddPackagesConfig' values not used. +-- Returns any of the 'AddPackagesConfig' values not used and also all +-- non-trivial package location completions. -- -- @since 0.1.0.0 addAndCompletePackagesToSnapshot :: (HasPantryConfig env, HasLogFunc env, HasProcessContext env) - => Utf8Builder + => RawSnapshotLocation -- ^ Text description of where these new packages are coming from, for error -- messages only + -> Map RawPackageLocationImmutable PackageLocationImmutable -- ^ Cached data from snapshot lock file -> [RawPackageLocationImmutable] -- ^ new packages -> AddPackagesConfig -> Map PackageName SnapshotPackage -- ^ packages from parent -> RIO env (Map PackageName SnapshotPackage, [CompletedPLI], AddPackagesConfig) -addAndCompletePackagesToSnapshot source newPackages (AddPackagesConfig drops flags hiddens options) old = do - let addPackage (ps, completed) loc = do - name <- getPackageLocationName loc - loc' <- completePackageLocation loc +addAndCompletePackagesToSnapshot loc cachedPL newPackages (AddPackagesConfig drops flags hiddens options) old = do + let source = display loc + addPackage :: (HasPantryConfig env, HasLogFunc env, HasProcessContext env) + => ([(PackageName, SnapshotPackage)],[CompletedPLI]) + -> RawPackageLocationImmutable + -> RIO env ([(PackageName, SnapshotPackage)], [CompletedPLI]) + addPackage (ps, completed) rawLoc = do + name <- getPackageLocationName rawLoc + complLoc <- cachedSnapshotCompletePackageLocation cachedPL rawLoc let p = (name, SnapshotPackage - { spLocation = loc' + { spLocation = complLoc , spFlags = Map.findWithDefault mempty name flags , spHidden = Map.findWithDefault False name hiddens , spGhcOptions = Map.findWithDefault [] name options }) - if toRawPLI loc' == loc - then pure (p:ps, completed) - else pure (p:ps, (loc, loc'):completed) + completed' = if toRawPLI complLoc == rawLoc + then completed + else (rawLoc, complLoc):completed + pure (p:ps, completed') (revNew, revCompleted) <- foldM addPackage ([], []) newPackages let (newSingles, newMultiples) = partitionEithers @@ -1194,20 +1223,19 @@ addAndCompletePackagesToSnapshot source newPackages (AddPackagesConfig drops fla loadRawSnapshotLayer :: (HasPantryConfig env, HasLogFunc env) => RawSnapshotLocation - -> RIO env (Either WantedCompiler (RawSnapshotLayer, SHA256)) -- FIXME remove SHA? Be smart? + -> RIO env (Either WantedCompiler (RawSnapshotLayer, CompletedSL)) loadRawSnapshotLayer (RSLCompiler compiler) = pure $ Left compiler -loadRawSnapshotLayer sl@(RSLUrl url blob) = - handleAny (throwIO . InvalidSnapshot sl) $ do +loadRawSnapshotLayer rsl@(RSLUrl url blob) = + handleAny (throwIO . InvalidSnapshot rsl) $ do bs <- loadFromURL url blob value <- Yaml.decodeThrow bs - snapshot <- warningsParserHelperRaw sl value Nothing - pure $ Right (snapshot, SHA256.hashBytes bs) -loadRawSnapshotLayer sl@(RSLFilePath fp) = - handleAny (throwIO . InvalidSnapshot sl) $ do + snapshot <- warningsParserHelperRaw rsl value Nothing + pure $ Right (snapshot, (rsl, SLUrl url (bsToBlobKey bs))) +loadRawSnapshotLayer rsl@(RSLFilePath fp) = + handleAny (throwIO . InvalidSnapshot rsl) $ do value <- Yaml.decodeFileThrow $ toFilePath $ resolvedAbsolute fp - sha <- SHA256.hashFile $ toFilePath $ resolvedAbsolute fp - snapshot <- warningsParserHelperRaw sl value $ Just $ parent $ resolvedAbsolute fp - pure $ Right (snapshot, sha) + snapshot <- warningsParserHelperRaw rsl value $ Just $ parent $ resolvedAbsolute fp + pure $ Right (snapshot, (rsl, SLFilePath fp)) -- | Parse a 'SnapshotLayer' value from a 'SnapshotLocation'. -- @@ -1219,20 +1247,19 @@ loadRawSnapshotLayer sl@(RSLFilePath fp) = loadSnapshotLayer :: (HasPantryConfig env, HasLogFunc env) => SnapshotLocation - -> RIO env (Either WantedCompiler (RawSnapshotLayer, SHA256)) -- FIXME remove SHA? Be smart? + -> RIO env (Either WantedCompiler RawSnapshotLayer) loadSnapshotLayer (SLCompiler compiler) = pure $ Left compiler loadSnapshotLayer sl@(SLUrl url blob) = handleAny (throwIO . InvalidSnapshot (toRawSL sl)) $ do bs <- loadFromURL url (Just blob) value <- Yaml.decodeThrow bs snapshot <- warningsParserHelper sl value Nothing - pure $ Right (snapshot, SHA256.hashBytes bs) + pure $ Right snapshot loadSnapshotLayer sl@(SLFilePath fp) = handleAny (throwIO . InvalidSnapshot (toRawSL sl)) $ do value <- Yaml.decodeFileThrow $ toFilePath $ resolvedAbsolute fp - sha <- SHA256.hashFile $ toFilePath $ resolvedAbsolute fp snapshot <- warningsParserHelper sl value $ Just $ parent $ resolvedAbsolute fp - pure $ Right (snapshot, sha) + pure $ Right snapshot loadFromURL :: (HasPantryConfig env, HasLogFunc env) diff --git a/subs/pantry/src/Pantry/Types.hs b/subs/pantry/src/Pantry/Types.hs index 2beb03c2de..3dd8be058a 100644 --- a/subs/pantry/src/Pantry/Types.hs +++ b/subs/pantry/src/Pantry/Types.hs @@ -90,6 +90,7 @@ module Pantry.Types , RawSnapshotLocation (..) , SnapshotLocation (..) , toRawSL + , parseHackageText , parseRawSnapshotLocation , RawSnapshotLayer (..) , SnapshotLayer (..) @@ -105,6 +106,7 @@ module Pantry.Types , cabalFileName , SnapshotCacheHash (..) , getGlobalHintsFile + , bsToBlobKey ) where import RIO @@ -504,6 +506,16 @@ instance Display Repo where (if T.null subdir then mempty else " in subdirectory " <> display subdir) +instance FromJSON Repo where + parseJSON = + withObject "Repo" $ \o -> do + repoSubdir <- o .: "subdir" + repoCommit <- o .: "commit" + (repoType, repoUrl) <- + (o .: "git" >>= \url -> pure (RepoGit, url)) <|> + (o .: "hg" >>= \url -> pure (RepoHg, url)) + pure Repo {..} + -- An unexported newtype wrapper to hang a 'FromJSON' instance off of. Contains -- a GitHub user and repo name separated by a forward slash, e.g. "foo/bar". @@ -710,6 +722,32 @@ instance FromJSON PackageIdentifierRevision where Left e -> fail $ show e Right pir -> pure pir +-- | Parse a hackage text. +parseHackageText :: Text -> Either PantryException (PackageIdentifier, BlobKey) +parseHackageText t = maybe (Left $ PackageIdentifierRevisionParseFail t) Right $ do + let (identT, cfiT) = T.break (== '@') t + PackageIdentifier name version <- parsePackageIdentifier $ T.unpack identT + (csha, csize) <- + case splitColon cfiT of + Just ("@sha256", shaSizeT) -> do + let (shaT, sizeT) = T.break (== ',') shaSizeT + sha <- either (const Nothing) Just $ SHA256.fromHexText shaT + msize <- + case T.stripPrefix "," sizeT of + Nothing -> Nothing + Just sizeT' -> + case decimal sizeT' of + Right (size', "") -> Just $ (sha, FileSize size') + _ -> Nothing + pure msize + _ -> Nothing + pure $ (PackageIdentifier name version, BlobKey csha csize) + +splitColon :: Text -> Maybe (Text, Text) +splitColon t' = + let (x, y) = T.break (== ':') t' + in (x, ) <$> T.stripPrefix ":" y + -- | Parse a 'PackageIdentifierRevision' -- -- @since 0.1.0.0 @@ -737,10 +775,6 @@ parsePackageIdentifierRevision t = maybe (Left $ PackageIdentifierRevisionParseF Nothing -> pure CFILatest _ -> Nothing pure $ PackageIdentifierRevision name version cfi - where - splitColon t' = - let (x, y) = T.break (== ':') t' - in (x, ) <$> T.stripPrefix ":" y data Mismatch a = Mismatch { mismatchExpected :: !a @@ -1377,6 +1411,18 @@ instance Display PackageMetadata where , "cabal file == " <> display (pmCabal pm) ] +instance FromJSON PackageMetadata where + parseJSON = + withObject "PackageMetadata" $ \o -> do + pmCabal :: BlobKey <- o .: "cabal-file" + pantryTree :: BlobKey <- o .: "pantry-tree" + CabalString pkgName <- o .: "name" + CabalString pkgVersion <- o .: "version" + let pmTreeKey = TreeKey pantryTree + pmIdent = PackageIdentifier {..} + pure PackageMetadata {..} + + -- | Conver package metadata to its "raw" equivalent. -- -- @since 0.1.0.0 @@ -1489,6 +1535,54 @@ rpmToPairs (RawPackageMetadata mname mversion mtree mcabal) = concat , maybe [] (\cabal -> ["cabal-file" .= cabal]) mcabal ] +instance FromJSON (WithJSONWarnings (Unresolved PackageLocationImmutable)) where + parseJSON v = repoObject v <|> archiveObject v <|> hackageObject v <|> github v + <|> fail ("Could not parse a UnresolvedPackageLocationImmutable from: " ++ show v) + where + repoObject :: Value -> Parser (WithJSONWarnings (Unresolved PackageLocationImmutable)) + repoObject value = do + pm <- parseJSON value + repo <- parseJSON value + pure $ noJSONWarnings $ pure $ PLIRepo repo pm + + archiveObject value = do + pm <- parseJSON value + withObjectWarnings "UnresolvedPackageLocationImmutable.PLIArchive" (\o -> do + Unresolved mkArchiveLocation <- parseArchiveLocationObject o + archiveHash <- o ..: "sha256" + archiveSize <- o ..: "size" + archiveSubdir <- o ..:? "subdir" ..!= "" + pure $ Unresolved $ \mdir -> do + archiveLocation <- mkArchiveLocation mdir + pure $ PLIArchive Archive {..} pm + ) value + + hackageObject value = + withObjectWarnings "UnresolvedPackagelocationimmutable.PLIHackage (Object)" (\o -> do + treeKey <- o ..: "pantry-tree" + htxt <- o ..: "hackage" + case parseHackageText htxt of + Left e -> fail $ show e + Right (pkgIdentifier, blobKey) -> + pure $ pure $ PLIHackage pkgIdentifier blobKey (TreeKey treeKey)) value + + github value = do + pm <- parseJSON value + withObjectWarnings "UnresolvedPackagelocationimmutable.PLIArchive:github" (\o -> do + GitHubRepo ghRepo <- o ..: "github" + commit <- o ..: "commit" + let archiveLocation = ALUrl $ T.concat + [ "https://github.com/" + , ghRepo + , "/archive/" + , commit + , ".tar.gz" + ] + archiveHash <- o ..: "sha256" + archiveSize <- o ..: "size" + archiveSubdir <- o ..: "subdir" + pure $ pure $ PLIArchive Archive {..} pm) value + instance FromJSON (WithJSONWarnings (Unresolved (NonEmpty RawPackageLocationImmutable))) where parseJSON v = http v @@ -1497,7 +1591,7 @@ instance FromJSON (WithJSONWarnings (Unresolved (NonEmpty RawPackageLocationImmu <|> repo v <|> archiveObject v <|> github v - <|> fail ("Could not parse a UnresolvedPackageLocationImmutable from: " ++ show v) + <|> fail ("Could not parse a UnresolvedRawPackageLocationImmutable from: " ++ show v) where http :: Value -> Parser (WithJSONWarnings (Unresolved (NonEmpty RawPackageLocationImmutable))) http = withText "UnresolvedPackageLocationImmutable.UPLIArchive (Text)" $ \t -> @@ -1910,6 +2004,27 @@ instance NFData SnapshotLocation instance ToJSON SnapshotLocation where toJSON sl = toJSON (toRawSL sl) +instance FromJSON (WithJSONWarnings (Unresolved SnapshotLocation)) where + parseJSON v = file v <|> url v <|> compiler v + where + file = withObjectWarnings "SLFilepath" $ \o -> do + ufp <- o ..: "filepath" + pure $ Unresolved $ \mdir -> + case mdir of + Nothing -> throwIO $ InvalidFilePathSnapshot ufp + Just dir -> do + absolute <- resolveFile dir (T.unpack ufp) + let fp = ResolvedPath (RelFilePath ufp) absolute + pure $ SLFilePath fp + url = withObjectWarnings "SLUrl" $ \o -> do + url' <- o ..: "url" + sha <- o ..: "sha256" + size <- o ..: "size" + pure $ Unresolved $ \_ -> pure $ SLUrl url' (BlobKey sha size) + compiler = withObjectWarnings "SLCompiler" $ \o -> do + c <- o ..: "compiler" + pure $ Unresolved $ \_ -> pure $ SLCompiler c + -- | Convert snapshot location to its "raw" equivalent. -- -- @since 0.1.0.0 @@ -2151,3 +2266,10 @@ getGlobalHintsFile = do root <- view $ pantryConfigL.to pcRootDir globalHintsRelFile <- parseRelFile "global-hints-cache.yaml" pure $ root globalHintsRelFile + +-- | Creates BlobKey for an input ByteString +-- +-- @sinc 0.1.0.0 +bsToBlobKey :: ByteString -> BlobKey +bsToBlobKey bs = + BlobKey (SHA256.hashBytes bs) (FileSize (fromIntegral (B.length bs))) diff --git a/subs/pantry/test/Pantry/TypesSpec.hs b/subs/pantry/test/Pantry/TypesSpec.hs index c45898ed26..2de9e75ee0 100644 --- a/subs/pantry/test/Pantry/TypesSpec.hs +++ b/subs/pantry/test/Pantry/TypesSpec.hs @@ -1,22 +1,36 @@ {-# LANGUAGE NoImplicitPrelude #-} +{-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE QuasiQuotes #-} {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE FlexibleContexts #-} -module Pantry.TypesSpec (spec) where +{-# LANGUAGE FlexibleInstances #-} -import Test.Hspec +module Pantry.TypesSpec + ( spec + ) where + +import Data.Aeson.Extended +import qualified Data.ByteString.Char8 as S8 +import qualified Data.Yaml as Yaml +import Distribution.Types.PackageName (mkPackageName) +import Distribution.Types.Version (mkVersion) import Hedgehog import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range import Pantry +import Pantry.Internal + ( Tree(..) + , TreeEntry(..) + , mkSafeFilePath + , parseTree + , renderTree + ) import qualified Pantry.SHA256 as SHA256 -import Pantry.Internal (parseTree, renderTree, Tree (..), TreeEntry (..), mkSafeFilePath) import RIO -import Distribution.Types.Version (mkVersion) import qualified RIO.Text as T -import qualified Data.Yaml as Yaml -import Data.Aeson.Extended (WithJSONWarnings (..)) -import qualified Data.ByteString.Char8 as S8 +import Test.Hspec +import Text.RawString.QQ import RIO.Time (Day (..)) hh :: HasCallStack => String -> Property -> Spec @@ -30,6 +44,22 @@ genBlobKey = BlobKey <$> genSha256 <*> (FileSize <$> (Gen.word (Range.linear 1 1 genSha256 :: Gen SHA256 genSha256 = SHA256.hashBytes <$> Gen.bytes (Range.linear 1 500) +samplePLIRepo :: ByteString +samplePLIRepo = + [r| +subdir: wai +cabal-file: + size: 1765 + sha256: eea52c4967d8609c2f79213d6dffe6d6601034f1471776208404781de7051410 +name: wai +version: 3.2.1.2 +git: https://github.com/yesodweb/wai.git +pantry-tree: + size: 714 + sha256: ecfd0b4b75f435a3f362394807b35e5ef0647b1a25005d44a3632c49db4833d2 +commit: d11d63f1a6a92db8c637a8d33e7953ce6194a3e0 +|] + spec :: Spec spec = do describe "WantedCompiler" $ do @@ -110,3 +140,56 @@ spec = do liftIO $ Yaml.toJSON (nightlySnapshotLocation day) `shouldBe` Yaml.String (T.pack $ "nightly-" ++ show day) + it "FromJSON instance for Repo" $ do + repValue <- + case Yaml.decodeThrow samplePLIRepo of + Just x -> pure x + Nothing -> fail "Can't parse Repo" + let repoValue = + Repo + { repoSubdir = "wai" + , repoType = RepoGit + , repoCommit = + "d11d63f1a6a92db8c637a8d33e7953ce6194a3e0" + , repoUrl = "https://github.com/yesodweb/wai.git" + } + repValue `shouldBe` repoValue + it "FromJSON instance for PackageMetadata" $ do + pkgMeta <- + case Yaml.decodeThrow samplePLIRepo of + Just x -> pure x + Nothing -> fail "Can't parse Repo" + let cabalSha = + SHA256.fromHexBytes + "eea52c4967d8609c2f79213d6dffe6d6601034f1471776208404781de7051410" + pantrySha = + SHA256.fromHexBytes + "ecfd0b4b75f435a3f362394807b35e5ef0647b1a25005d44a3632c49db4833d2" + (csha, psha) <- case (cabalSha, pantrySha) of + (Right csha, Right psha) -> pure (csha, psha) + _ -> fail "Failed decoding sha256" + let pkgValue = + PackageMetadata + { pmIdent = + PackageIdentifier + (mkPackageName "wai") + (mkVersion [3, 2, 1, 2]) + , pmTreeKey = TreeKey (BlobKey psha (FileSize 714)) + , pmCabal = BlobKey csha (FileSize 1765) + } + pkgMeta `shouldBe` pkgValue + it "parseHackageText parses" $ do + let txt = + "persistent-2.8.2@sha256:df118e99f0c46715e932fe82d787fc09689d87898f3a8b13f5954d25af6b46a1,5058" + hsha = + SHA256.fromHexBytes + "df118e99f0c46715e932fe82d787fc09689d87898f3a8b13f5954d25af6b46a1" + sha <- case hsha of + Right sha' -> pure sha' + _ -> fail "parseHackagetext: failed decoding the sha256" + let Right (pkgIdentifier, blobKey) = parseHackageText txt + blobKey `shouldBe` (BlobKey sha (FileSize 5058)) + pkgIdentifier `shouldBe` + PackageIdentifier + (mkPackageName "persistent") + (mkVersion [2, 8, 2]) diff --git a/test/integration/IntegrationSpec.hs b/test/integration/IntegrationSpec.hs index 786a3af8bf..c1ac7ed934 100644 --- a/test/integration/IntegrationSpec.hs +++ b/test/integration/IntegrationSpec.hs @@ -238,8 +238,15 @@ copyTree src dst = Just suffix <- return $ stripPrefix src srcfp let dstfp = dst stripHeadSeparator suffix createDirectoryIfMissing True $ takeDirectory dstfp - createSymbolicLink srcfp dstfp `catch` \(_ :: IOException) -> - copyFile srcfp dstfp -- for Windows + -- copying yaml files so lock files won't get created in + -- the source directory + if takeFileName srcfp /= "package.yaml" && + (takeExtensions srcfp == ".yaml" || takeExtensions srcfp == ".yml") + then + copyFile srcfp dstfp + else + createSymbolicLink srcfp dstfp `catch` \(_ :: IOException) -> + copyFile srcfp dstfp -- for Windows stripHeadSeparator :: FilePath -> FilePath stripHeadSeparator [] = [] diff --git a/test/integration/tests/lock-files/Main.hs b/test/integration/tests/lock-files/Main.hs new file mode 100644 index 0000000000..8f7c89700f --- /dev/null +++ b/test/integration/tests/lock-files/Main.hs @@ -0,0 +1,17 @@ +import Control.Monad (unless, when) +import Data.List (isInfixOf) +import StackTest +import System.Directory + +main :: IO () +main = do + copyFile "stack-2-extras" "stack.yaml" + stack ["build"] + lock1 <- readFile "stack.yaml.lock" + unless ("acme-dont" `isInfixOf` lock1) $ + error "Package acme-dont wasn't found in Stack lock file" + copyFile "stack-1-extra" "stack.yaml" + stack ["build"] + lock2 <- readFile "stack.yaml.lock" + when ("acme-dont" `isInfixOf` lock2) $ + error "Package acme-dont shouldn't be in Stack lock file anymore" diff --git a/test/integration/tests/lock-files/files/Lib.hs b/test/integration/tests/lock-files/files/Lib.hs new file mode 100644 index 0000000000..a3b82e6e83 --- /dev/null +++ b/test/integration/tests/lock-files/files/Lib.hs @@ -0,0 +1,2 @@ +foo :: Int +foo = 42 diff --git a/test/integration/tests/lock-files/files/package.yaml b/test/integration/tests/lock-files/files/package.yaml new file mode 100644 index 0000000000..36e02ec5e7 --- /dev/null +++ b/test/integration/tests/lock-files/files/package.yaml @@ -0,0 +1,4 @@ +name: example +library: + dependencies: + - base diff --git a/test/integration/tests/lock-files/files/stack-1-extra b/test/integration/tests/lock-files/files/stack-1-extra new file mode 100644 index 0000000000..94527115ec --- /dev/null +++ b/test/integration/tests/lock-files/files/stack-1-extra @@ -0,0 +1,3 @@ +resolver: lts-11.22 +extra-deps: +- acme-cuteboy-0.1.0.0 diff --git a/test/integration/tests/lock-files/files/stack-2-extras b/test/integration/tests/lock-files/files/stack-2-extras new file mode 100644 index 0000000000..5415f52ee4 --- /dev/null +++ b/test/integration/tests/lock-files/files/stack-2-extras @@ -0,0 +1,4 @@ +resolver: lts-11.22 +extra-deps: +- acme-cuteboy-0.1.0.0 +- acme-dont-1.1