summaryrefslogtreecommitdiff
path: root/Tools (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Make sure we don't add ${WRKDIR}.Maxim Sobolev2008-10-211-4/+4
| | | | Notes: svn path=/head/; revision=221883
* Handle the rest of the edge cases; drop 5 as obsolete; cleanup.Mark Linimon2008-09-181-23/+35
| | | | | | | | | Now re-enabled on production. Approved by: portmgr (self) Notes: svn path=/head/; revision=220428
* Bring this into the new portbuild world order. While here, try toMark Linimon2008-09-171-27/+94
| | | | | | | | | suppress hrefs for non-existant files (incomplete). Approved by: portmgr (self) Notes: svn path=/head/; revision=220421
* RELENG_5 is desupported.Mark Linimon2008-09-121-3/+3
| | | | | | | Approved by: portmgr (self) Notes: svn path=/head/; revision=220355
* Major overhaul:Mark Linimon2008-09-071-349/+172
| | | | | | | | | | | | - bring this closer to the default FreeBSD page style - remove unsupported releases - remove the date stamps, which no longer work - remove obsolete commented-out junk Discussed on: portmgr, some time ago Notes: svn path=/head/; revision=220059
* Add a new script for performing high-level administration of a build.Kris Kennaway2008-08-081-518/+496
| | | | | | | | | | | | | | | | | | list : lists available builds clone : creates a new build by cloning a previous one portsupdate : update a ports tree to the latest ZFS snapshot srcupdate : update a src tree to the latest ZFS snapshot cleanup : clean up or remove a build on the clients destroy : remove a build on the server There is some trickiness here in that various commands either expect to run as root, or expect to run as a ports-* user. For the latter case we can easily use su to proxy as the ports user when running as root; for the former we use the buildproxy to validate and re-execute the command as root. Notes: svn path=/head/; revision=218250
* Add an option to clean up old distfiles. We don't have enough space toKris Kennaway2008-08-081-0/+10
| | | | | | | keep dozens of full distfile sets lying around. Notes: svn path=/head/; revision=218249
* Update the list of openoffice.org ports: remove 1.x and add 3.xErwin Lansing2008-07-291-1/+1
| | | | Notes: svn path=/head/; revision=217823
* Oops, commit the right version of this (adds extra checks that onlyKris Kennaway2008-07-261-0/+5
| | | | | | | expected commands will be proxied) Notes: svn path=/head/; revision=217607
* * Simple client that passes on a build request to the buildproxyKris Kennaway2008-07-261-0/+32
| | | | Notes: svn path=/head/; revision=217606
* * Python daemon run as root that proxies privileged build commands forKris Kennaway2008-07-261-0/+75
| | | | | | | | | | | the ports-* users. Currently it is not possible to delegate management of ZFS filesystems to non-root users, so root privilege is required to manipulate them. We validate the command passed on a local domain socket and re-execute the build script with the requested parameters. Notes: svn path=/head/; revision=217605
* Script run from cron to regularly update the master ZFS copies of theKris Kennaway2008-07-261-0/+63
| | | | | | | | | | | | | | | | | ports and source trees. Since we have >=1 consumer of these trees that run frequently but do not insist on up-to-the-second trees, it makes sense to "pre-update" them regularly and then then re-use in all of the consumers, instead of potentially doing several updates simultaneously or on demand. Consumers can clone the ZFS snapshot into their local filesystem which takes a couple of seconds instead of minutes or tens of minutes for the CVS update. We update to a date stamp instead of "." because this avoids ambiguity of commits that happen while the tree update is in progress (unfortunately it's slower). Notes: svn path=/head/; revision=217604
* Script run from cron on the package clients to report metrics to ganglia.Kris Kennaway2008-07-261-0/+35
| | | | | | | | | | | Currently we collect: * The current and maximum number of vnodes in use * The number of packages built over the past hour Notes: svn path=/head/; revision=217603
* Simple script to expire ZFS snapshots older than a certain ageKris Kennaway2008-07-261-0/+47
| | | | Notes: svn path=/head/; revision=217602
* Python script for backing up ZFS filesystems on pointyhat. For eachKris Kennaway2008-07-261-0/+217
| | | | | | | | | listed filesystem we take a new snapshot each time it is run and if the last full backup was not too long ago, do a compressed incremental backup from the previous backup. Notes: svn path=/head/; revision=217601
* * Add comment that this is unusedKris Kennaway2008-07-261-0/+1
| | | | Notes: svn path=/head/; revision=217600
* * Add commentsKris Kennaway2008-07-261-1/+6
| | | | Notes: svn path=/head/; revision=217599
* * CleanupKris Kennaway2008-07-261-28/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Support a meta-hostname of 'all' for setting up all clients at once. This is better than the old way of running one copy of the script for each client by hand, since it is easier and involves less duplicated work. * We copy in the per-build ports, src, and bindist .tbz files and .md5 checksums, as well as refreshing the build scripts and bindist-$(hostname).tar customization tarball. * The -force switch forces copying of files and re-extraction of the tarballs on the client. This is necessary in order to propagate local changes to the tarballs after the initial client setup (e.g. if you need to change a file in the ports tree, it must be recompressed, redistributed, and re-extracted on the client). * The -queue switch will poll the client's job queue after completion of the setup. This is racy and should only be used when the machine is not currently accepting jobs. * For cleaning up a build the 'build cleanup' command should now be used instead. It calls back into this command but also allows full clenaup of build-local files on the client. TODO: "all" setups are hard on the server since they may spawn dozens of rsyncs at once. A better solution would be to have a worker pool of setup tasks to limit the maximum load. Notes: svn path=/head/; revision=217598
* * CleanupKris Kennaway2008-07-261-28/+23
| | | | | | | | | | | | | * Catch up to build ID directory changes * Make it easier to kill a build by not running dopackages in the background where it is detached from shell job control. Now, sending a termination signal to this process (e.g. ^C) will also kill off the dopackages script and in turn the processes created by it. Some background processes spawned by dopackages, pdispatch, etc, may still remain and need to be killed by hand. Notes: svn path=/head/; revision=217597
* * CleanupKris Kennaway2008-07-261-496/+581
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Improve usage() * Fix a variety of small bugs * Remove support for -ftp builds: we have not supported direct uploading for many years due to the desire to manually inspect build output for quality * All data associated to a build is now localized in its own directory named according to a build ID: /var/portbuild/${arch}/${branch}/builds/${buildid}, where ${buildid} is the creation time. These are actually ZFS filesystems. * Tasks such as cloning a new build, updating a ZFS snapshot, and cleaning up a build are exported to the "build" script, which can be used independently. * Creating a new build is done by ZFS cloning and takes a couple of seconds since it is copy-on-write (i.e. no data needs to be copied). * Ports and source trees are also cloned from pre-updated ZFS images (updated regularly from the "updatesnap" cron job). In most cases we do not care if we are building a ports tree that is an hour or so old since it will become outdated almost immediately anyway, so no matter what we do there will be times when a port has been fixed by the time the build error is generated by a client. * In case an up-to-the-second tree is desired, the -portscvs and -srccvs switches update the existing ports tree via CVS. * -noports and -nosrc can be used to prevent any automatic changes to the ports tree. This is useful for dealing with local modifications (e.g. for -exp builds), since the default when creating a new build is to replace the previous trees with fresh, pristine trees. If you forget to use this then any local changes that are not also present in other trees will be lost. * By default we keep two builds for each arch/branch pair. These build IDs also may be referred to via "latest" and "previous" symlinks. When creating a new build, the old "previous" build is destroyed by default, unless it was originally created using the -keep switch. This prevents the build from being destroyed automatically. * By default when a build finishes all of the clients are completely cleaned up (i.e. all build data such as ports trees, tarballs, client chroots, etc are deleted). This is needed to save space on the clients. If you expect to *immediately* perform further builds after this one completes, the -nocleanup switch prevents this step. Otherwise they will just be set up again if further builds are scheduled. * Try to parallelize build pre-processing as much as possible, by running jobs in the background wherever possible. In several places we operate on the same parts of the filesystem from multiple jobs, so we can make good use of caching to improve performance * Clients no longer need to be set up explicitly at the start of the build, they will be set up on-demand when the first job is dispatched to them. This allows fast clients or those that already have been set up to begin building ports as soon as possible, while slow clients are set up in the background. It also improves robustness of client recovery, e.g. if the client was offline at the time of build startup but later brought back online. * Optimize copying back in the previous set of restricted packages by hardlinking instead of copying. TODO: The record of failed ports is arch/branch-global still. This is the only thing preventing us from running concurrent builds of the same arch/branch (e.g. while one is stuck building openoffice, the next build can start to keep the cluster busy). The difficulty is that one build from a later ports tree may signal that a build was successful, then a phase 2 build from an earlier ports tree may indicate that it was broken. The solution is probably to migrate this to a real database instead of a flat file, and query it for the set of broken ports as of a certain ports tree date. Notes: svn path=/head/; revision=217596
* * Catch up to build ID directory changesKris Kennaway2008-07-261-42/+155
| | | | | | | | | | | | | | | | | | | | | * Clients no longer mount ports/src trees via NFS (even the FreeBSD.org local clients). This was putting too much load on the server and slowing down builds. * Instead ports and src .tbz files are pushed to the clients and unpacked. MD5 checksums are used to verify correctness * -force forces re-extraction of the tarballs even if they exist and appear to be checked out * Also unpack the compressed bindist TODO: When we are not using md or ZFS builds it would be even faster to keep an unpacked copy of the bindist on the scratch filesystem and hardlink the files into the target directory Notes: svn path=/head/; revision=217594
* * Catch up to build ID directory changesKris Kennaway2008-07-261-3/+24
| | | | | | | | * Optimize by copying old packages using cpio -dumpl (i.e. create hardlink instead of copying the file). Notes: svn path=/head/; revision=217592
* * Catch up to build ID directory changesKris Kennaway2008-07-262-9/+16
| | | | Notes: svn path=/head/; revision=217591
* * CleanupKris Kennaway2008-07-261-50/+51
| | | | | | | | * Catch up to build ID directory changes * Remove need for /etc/arch file Notes: svn path=/head/; revision=217590
* * CleanupKris Kennaway2008-07-261-71/+73
| | | | | | | | | * Catch up to build ID directory changes * Improved support for ZFS * Desupport X11BASE Notes: svn path=/head/; revision=217589
* * CleanupKris Kennaway2008-07-261-75/+120
| | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Improved support for ZFS builds * Improved robustness * Report status verbosely to the caller; whether we succeeded in claiming a chroot, whether the caller needs to first set up the client, or whether a setup is in progress. * If we discover that the client has not been set up either because it freshly booted and newfs'ed its filesystem, or because a particular build has not yet been encountered, atomically claim a cookie and report this to the caller to act on Notes: svn path=/head/; revision=217588
* * CleanupKris Kennaway2008-07-263-24/+50
| | | | | | | * Catch up to build ID directory changes Notes: svn path=/head/; revision=217587
* * CleanupKris Kennaway2008-07-261-19/+74
| | | | | | | | | | * Catch up to build ID directory changes * Add helper functions for resolving a build ID symlink and validating an arch/branch combination (centralize instead of doing it in many scripts) Notes: svn path=/head/; revision=217586
* * Desupport alpha and ia64Kris Kennaway2008-07-261-4/+3
| | | | | | | * Catch up to build ID directory changes Notes: svn path=/head/; revision=217585
* * Add commentsKris Kennaway2008-07-261-2/+2
| | | | Notes: svn path=/head/; revision=217584
* * Implement basename and dirname using shell builtinsKris Kennaway2008-07-261-0/+8
| | | | Notes: svn path=/head/; revision=217583
* * CleanupKris Kennaway2008-07-261-13/+15
| | | | | | | | * Catch up to build ID directory changes * Record package build completion for reporting to ganglia Notes: svn path=/head/; revision=217582
* * CleanupKris Kennaway2008-07-261-104/+160
| | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Add support for ssh_cmd and scp_cmd to allow using HPN-SSH with the none cipher where possible (for performance) * Lazy client setup; claim-chroot will report if the client needs to be set up with this buildid, and we initiate the setup and poll until it is complete. This allows fast clients to begin building before slow ones have finished setting up. TODO: a better solution would be to avoid trying to dispatch jobs onto clients that are in the process of setting up, since they often have low loads and are picked preferentially by the job scheduler. Notes: svn path=/head/; revision=217580
* * CleanupKris Kennaway2008-07-261-65/+41
| | | | | | | | | | | | | * Remove vestiges of archaic support for building bindists from FTP snapshots; we haven't used this for years and building a world is no longer a challenge * Revert half-baked bindist generation number and make it per-buildid instead. Compress and md5 it for distribution to the clients. TODO: Merge with makeworld? Notes: svn path=/head/; revision=217576
* * CleanupKris Kennaway2008-07-261-17/+24
| | | | | | | | | * Catch up to build ID directory changes * Optimize by using ECHO_MSG=true instead of /usr/bin/true * Try harder to avoid pollution from local host Notes: svn path=/head/; revision=217574
* * CleanupKris Kennaway2008-07-261-7/+18
| | | | | | | * Catch up to build ID directory layout Notes: svn path=/head/; revision=217573
* * CleanupKris Kennaway2008-07-261-51/+18
| | | | | | | | | * Catch up to build ID directory changes * Export the INDEX_PRISTINE and INDEX_QUIET variables (old bug) * Desupport X11BASE Notes: svn path=/head/; revision=217572
* * CleanupKris Kennaway2008-07-261-38/+39
| | | | | | | | * Catch up to build ID directory changes * Desupport 5.x Notes: svn path=/head/; revision=217570
* * CleanupKris Kennaway2008-07-261-15/+17
| | | | | | | | * Catch up to build ID directory changes * Optimize by using __MAKE_SHELL=/rescue/sh Notes: svn path=/head/; revision=217569
* Rewrite in python and combine the functions of the formerKris Kennaway2008-07-261-35/+263
| | | | | | | | | | | | checkmachines script. Polls build machines for their status either once-off or regularly as a daemon. Optionally it will update the queue entries but this remains subject to race conditions. TODO: Integrate with queue manager and forward machine status changes to it Notes: svn path=/head/; revision=217568
* More verbose status reporting using key=value format. We now alsoKris Kennaway2008-07-261-8/+18
| | | | | | | | report error status, architecture and OS version, and available build environments, as well as load and number of running jobs Notes: svn path=/head/; revision=217567
* Major optimizations for 'make index' and other recursive traversalKris Kennaway2008-07-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | targets. * Use /rescue/sh for index builds instead of /bin/sh, when it exists. The former is statically linked and faster to execute, which becomes significant when executing it tens of thousands of times. This trick can be used with other recursive targets by passing in __MAKE_SHELL. * Get rid of make variable assignments that use != command invocations in the critical path, using several methods: - rewriting logic to use shell or make builtins instead of external command executions - macroizing commands and executing them in the targets where they are needed instead of with every invocation of make - precomputing the results of invariant commands in bsd.port.subdir.mk and passing them in explicitly to child makes, and using this to avoid recalculation in all the children. NB: the commands are still run one per top-level subdirectory but this does not currently seem to be a major issue. They could be moved further up into the top-level Makefile at the cost of some cleanliness. - Committers are strongly discouraged from adding further "bare" != assignments to the ports tree, even in their own ports. One of the above strategies should be used to avoid future bloat. * Rewrite the core 'describe' target to work entirely within a single shell process using only builtin commands. The old version is retained as a backup for use on systems older than 603104, which does not have the make :u modifier. This cuts down the number of processes executed during the course of a 'make index' by an order of magnitude, and we are essentially now amortized to the minimum of a single make + sh instance per port, plus whatever commands the port makefile itself executes (which are usually unnecessary and bogus). * Less validation of the WWW: target is performed; this can become policed at a port level by portlint. Specifically we look at the second word of the first line beginning with "WWW:" in pkg-descr, and append "http://" to it unless it already begins with "http://", "https://" or "ftp://". Thanks to dougb for the idea of how to extract WWW: using shell builtins. * Use the "true" shell builtin instead of echo > /dev/null for a measurable decrease in CPU use. * Add a note about dubious escaping strategy in bsd.port.subdir.mk * Minor change in output of 'make describe': it no longer strips trailing CR characters from pkg-descr files with MSDOS CR/LF termination. Instead the makeindex perl script that post-processes make describe into the INDEX is tweaked to strip on input. The bottom line is that on my test hardware INDEX builds are now faster by more than a factor of 2 and with a reduction in system time by a factor of 4-8 depending on configuration. Notes: svn path=/head/; revision=217132
* Have a different approach on how to check if the modules file has changed.Edwin Groothuis2008-07-141-9/+7
| | | | | | | | | | | | | Because the $FreeBSD$ keyword isn't expanded in the new version, we can't just do a diff, check the return value and ignore the output. Every new modules file, changed or not with regarding to the contents, has at least four lines in the diff output (line number, old line, seperator, new line). Only commit it if there are more than four lines difference between it. Notes: svn path=/head/; revision=216816
* Keep the sources of the CVSROOT-ports/modules update script safelyEdwin Groothuis2008-07-084-0/+361
| | | | | | | in the ports CVS repository. Notes: svn path=/head/; revision=216557
* [PATCH] Tools/scripts: Some cleanups, additions to bump_version.plEdwin Groothuis2008-07-061-28/+33
| | | | | | | | | | | | | | | | | - added -c, doesn't change anything - added -n, deal with an already checkouted tree - removed negative logic in favor of 'unless' - switch to 3 arg form of open() - don't use globs for filehandles, this is been obsolete since at least 5.6.1 - handle possible errors in close() - allow CVSROOT to be overriden in the ENV PR: ports/125025 Submitted by: "Philip M. Gollucci" <pgollucci@p6m7g8.com> Notes: svn path=/head/; revision=216390
* Reflect latest changes from production:Mark Linimon2008-07-021-39/+43
| | | | | | | | | - no more 5-exp - add 8, 8-exp - fix two error-name hrefs Notes: svn path=/head/; revision=216144
* This conversion script is no longer usefulKris Kennaway2008-06-251-74/+0
| | | | Notes: svn path=/head/; revision=215781
* CVSROOT/modules no longer contains an entry per port, remove supportErwin Lansing2008-06-194-300/+9
| | | | | | | for this from these scripts. Notes: svn path=/head/; revision=215312
* Modernize this script a bit.Kris Kennaway2008-06-111-13/+10
| | | | | | | | | | | | | * Remove 5.x support * Leave the archaic ftp snapshot support for now, it is not hurting anything but will not work * Be more careful when removing files (use absolute paths) * Switch to bindist/tmp for the tmp dir * Fix the recording of the bindist.tar generation number * Get rid of redundant or useless processing of the world image Notes: svn path=/head/; revision=214666
* * Distfile collection is now the default; replace -distfiles with -nodistfilesKris Kennaway2008-06-111-9/+13
| | | | | | | | | * Record the CVS update stamp in some extra places and make sure to remove it if the build is started with -noportscvs (since this probably means the ports tree was updated by hand at some random time) Notes: svn path=/head/; revision=214665
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-02 00:01:09 +0000